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ABSTRACT 



A watermarking procedure that is applicable to images, 
audio, video and multimedia data to be watermarked divides 
the data to be watermarked into a set of nxn blocks, such as 
the 8x8 blocks of MPEG. The same watermark signal can be 
distributed throughout the set of blocks in a large variety of 
ways. This allows the insertion algorithm to be changed 
without affecting the decoders. The decoding procedure first 
sums together the DCT coefficients of N sets of 8x8 blocks 
to form a set of N summed 8x8 blocks and then extracts the 
watermark from the summed block. Since the sum of the 
DCT blocks is equal to the DCT of the sum of the intensity 
blocks, efficient decoding can occur in both the spatial and 
frequency domains. The symmetric nature of the decoding 
process allows geometric distortions to be handled in the 
spatial domain and other signal distortions to be handled in 
the frequency domain. Moreover, insertion of a watermark 
signal into image data and the subsequent extraction of the 
watermark from watermarked image data which has been 
subject to distortion between the times of insertion and 
extraction involves the insertion of multiple watermarks 
designed to survive predefined distortions of the image data, 
such as panscan or letterbox mode transformations. 
Alternatively, a registration pattern in the image data, after 
the image data containing the registration pattern is subject 
to an unknown distortion, is used to compensate for distor- 
tion of the watermarked image data. 

10 Claims, 10 Drawing Sheets 
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ROBUST DIGITAL WATERMARKING 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

This is a conversion of provisional application Ser. No. 5 
60/090,532, filed Jun. 24, 1998. 

FIELD OF THE INVENTION 

The present invention relates to digital watermarking of 
data including image, video and multimedia data. 10 
Specifically, the invention relates to insertion and detection 
or extraction of embedded signals for purposes of 
watermarking, in which the insertion and detection proce- 
dures are applied to sums of sub regions of the data. When 
these subregions correspond to the 8x8 pixel blocks used for 15 
MPEG and JPEG compression and decompression, the 
watermarking procedure can be tightly coupled with these 
compression algorithms to achieve very significant savings 
in computation. The invention also relates to the insertion 
and detection of embedded signals for the purposes of 20 
watermarking, in which the watermarked data might have 
undergone distortion between the times of insertion and 
detection of the watermark. 

BACKGROUND OF THE INVENTION 25 

The proliferation of digitized media such as image, video 
and multimedia is creating a need for a security system that 
facilitates the identification of the source of the material. 

Content providers, i.e. owners of works in digital data 30 
form, have a need to embed signals into video/image/ 
multimedia data, which can subsequently be detected by 
software, and/or hardware devices for purposes of authen- 
tication of copyright ownership, and copy control and man- 
agement. 35 

For example, a coded signal might be inserted in data to 
indicate that the data should not be copied. The embedded 
signal should preserve the image fidelity, be robust to 
common signal transformations and resistant to tampering. 
In addition, consideration must be given to the data rate that 40 
can be provided by the system, though current requirements 
are relatively low — a few bits per frame. 

In U.S. patent application Ser. No. 08/534,894, filed Sep. 
28, 1995, entitled "Secure Spread Spectrum Watermarking 
for Multimedia Data", which is incorporated herein by 45 
reference, there was proposed a spread spectrum watermark- 
ing method which embedded a watermark signal into per- 
ceptually significant regions of an image for the purposes of 
identifying the content owner and/or possessor. A strength of 
this approach is that the watermark is very difficult to 50 
remove. In fact, this method only allows the watermark to be 
read if the original image or data is available for comparison. 
This is because the original spectrum of the watermark is 
shaped to that of the image through a non-linear multipli- 
cative procedure, and this spectral shaping must be removed 55 
prior to detection by matched filtering. In addition, the 
watermark is usually inserted into the N largest spectral 
coefficients, the ranking of which is not preserved after 
watermarking. This method does not allow software and 
hardware devices to directly read embedded signals without 60 
access to the original un watermarked material. 

In an article by Cox et al., entitled "Secured Spectrum 
Watermarking for Multimedia" available at http:// 
www.neci.nj.nec.con/tr/index.html (Technical Report No. 
95-10) spread spectrum watermarking is described which 65 
embeds a pseudo-random noise sequence into the digital 
data for watermarking purposes. 
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The above prior art watermark extraction methodology 
requires the original image spectrum be subtracted from the 
watermark image spectrum. This restricts the use of the 
method when there is no original image or original image 
spectrum available to the decoder. One application where 
this presents a significant difficulty is for third party device 
providers desiring to read embedded information for opera- 
tion or denying operation of such a device. 

In U.S. Pat. No. 5,319,735 by R. D. Preuss et al entitled 
"Embedded Signaling*' digital information is encoded to 
produce a sequence of code symbols. The sequence of code 
symbols is embedded in an audio signal by generating a 
corresponding sequence of spread spectrum code signals 
representing the sequence of code symbols. The frequency 
components of the code signal being essentially confined to 
a preselected signaling band lying within the bandwidth of 
the audio signal and successive segments of the code signal 
corresponds to successive code symbols in the sequence. 
The audio signal is continuously frequency analyzed over a 
frequency band encompassing the signaling band and the 
code signal is dynamically filtered as a function of the 
analysis to provide a modified code signal with frequency 
component levels which are, at each time instant, essentially 
a preselected proportion of the levels of the audio signal 
frequency components in corresponding frequency ranges. 
The modified code signal and the audio signal are combined 
to provide a composite audio signal in which the digital 
information is embedded. This component audio signal is 
then recorded on a recording medium or is otherwise sub- 
jected to a transmission channel. Two key elements of this 
process are the spectral shaping and spectral equalization 
that occur at the insertion and extraction stages, respectively, 
thereby allowing the embedded signal to be extracted with- 
out access to the unwatermarked original data. 

In U.S. patent application Ser. No. 08/708,331, filed Sep. 
4, 1996, entitled "A Spread Spectrum Watermark for 
Embedded Signaling" by Cox, and incorporated herein by 
reference, there is described a method for extracting a 
watermark of embedded data from watermarked images or 
video without using an original or unwatermarked version of 
the data. 

This method of watermarking an image or image data for 
embedded signaling requires that the DCT (discrete cosine 
transform) and its inverse of the entire image be computed. 
There are fast algorithms for computing the DCT in N log 
N time, where N is the number of pixels in the image. 
However, for N<=5 12x512, the computational requirement is 
still high, particularly if the encoding and extracting pro- 
cesses must occur at video rates, i.e. 30 frames per second. 
This method requires approximately 30 times the computa- 
tion needed for MPEG-II decompression. 

One possible way to achieve real-time video watermark- 
ing is to only watermark every Nth frame. However, content 
owners wish to protect each and every video frame. 
Moreover, if it is known which frames contain embedded 
signals, it is simple to remove those frames with no notice- 
able degradation in the video signal. 

An alternative option is to insert the watermark into nxn 
blocks of the image (subimages) where n«N. If the block 
size is chosen to be 8x8, i.e. the same size as that used for 
MPEG image compression, then it is possible to tightly 
couple the watermark insertion and extraction procedures to 
those of the MPEG compression and decompression algo- 
rithms. Considerable computational saving can then be 
achieved since the most expensive computations relate to the 
calculation of the DCT and its inverse and these steps are 
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already computed as part of the compression and decom- so that the summed DCT blocks perfectly match the required 

pression algorithm. The incremental cost of watermarking is watermark signal, even though the resulting image appears 

then very small, typically less than five percent of the identical to the original. 

computational requirements associated with MPEG. Finally, it is well known that some problems, such as 
U.S. patent application Ser. No. 08/715,953, filed Sep. 19, 5 modeling the human visual system, are best performed in the 
1996, entitled "Watermarking of Image Data Using MPEG/ frequency domain, where other problems such as geometric 
JPEG Coefficients" which is incorporated herein by transformations are more conveniently dealt with in the 
reference, advances this work by using MPEG/JPEG coef- spatial domain. Since the computational cost of decoding the 
ficients to encode the image data. watermark is now symmetric, it is possible to switch from 
U.S. patent application Ser. No. 08/746,022, filed Nov. 5, 30 spatial to frequency domains at will in order to correct for 
1996, entitled "Digital Watermarking", which is incorpo- various signal transformations that may corrupt the water- 
rated herein by references, describes storing watermark mark, 
information into subimages and extracting watermark infor- 

mation from subimages. SUMMARY OF THE INVENTION 

A review of watermarking is found in an article by Cox et 15 The present invention concerns a novel insertion method 

al., entitled "A review of watermarking and the importance which emp i oys a specific model of the human visual system 

of perceptual modeling" in Proc. of EI'97, vol. 30-16, Feb. which prov ides much better control over image fidelity. 

9-14, 1997. Tests have shown that it is possible to obtain large signals 

There have been several proposals to watermark MPEG 2Q (more than 15 standard deviations from 0 correlation) with 

video or JPEG compressed still images. In all cases, each images that are indistinguishable from their respective origi- 

8x8 DCT block is modified to contain the watermark or a nal images. 

portion thereof. Consequently, decoding of the watermark ^ mel h 0 d handles robustness against various types of 

requires that each 8x8 block be individually analyzed to attacks in ways thal are easy t0 relatc t0 the specific type of 

extract the watermark signal contained therein. The indi- 25 attack. 

vidual extracted signals may then be combined to form a ™ . . , . , , . . , , c . . 

& . j i The method is adaptable so that the model or the human 

composite watermark, which is then compared with known , . . . . . . « - , ... „ . 

, r . „ iL i j visual system and the techniques used for handling attacks 

watermarks. Because each block must be analyzed , v , , . ... t 1 . . . . * . . 

....... , . 4 , J , , can be changed later without having to change the detector. 

individually, an uncompressed image must be converted ™_ w . . 4 . A . ... ~ A . 

i_ i • Li * i jt^^ * u- u • The result is that it is possible to continue improving 

back to the block-based DCT representation, which is com- _ n t . . , nun r ,. , . , r .. .? 

putationally expensive. Thus, whOe the decoder may be 30 watermarking, particularly DVD (dig. al video disk) 

♦ *• 11 « • * • .u t\/^ti j . „ / watermarking, even after many detectors have been 

computationally efficient in the DCT dom am, extracting a . t „ , ™ P . . t ' t . w „ n « 

4 r i r • i j • • . . installed. This is analogous to the situation with MPEG 

watermark from the spatial domain is much more expensive, r . . , , ~ . , , . , ... 

„ „ . , . . video for which encoder technology can be improved with- 

To allow for computationally efficient detection of the out having tQ change existing decoders . 

watermark in both the spatial and DCT domains, a water- « . . ^ . . , „ . 

mark may be inserted in the sum of all the 8x8 blocks in the . , Us f. of * e ^ Dt . mse ,f> n aU ° ws a sim P' e 

DCTdomain,orthesun,ofasubsetofallthe8x8blocksin ^ onthm m ei,her or dec ° m P ressed 
the DCT domain, A major advantage of this approach is that 

if the image is only available in the spatial domain, then the ^ invention also concerns a novel detection method 

summation can also be performed in the spatial domain to 40 which * eas y t0 implement, easy to analyze and has a low 

compute a small set of summed 8x8 blocks and only those computational cost, whether the incoming video is MPEG 

blocks must then be transformed into the DCT domain. This compressed or uncompressed. 

is because the sum of the DCT blocks is equal to the DCT The present invention also concerns a novel insertion 

of the sum of the intensities. Thus, the computational cost of method that hides multiple patterns in the data. These 

decoding in the DCT and spatial domains is approximately 45 patterns fall into two categories: 1) registration patterns used 

the same. during detection to compensate for translational shifts, and 

A second advantage of watermarking the sum of the DCT 2 ) watermark patterns that encode the information content of 

blocks is that there are an unlimited number of equivalent tne watermark. 

methods to apportion the watermark throughout the image. A principal object of the present invention is the provision 

For example, if the watermark requires a change of Ai to the 50 of a digital watermark insertion method which allows detec- 

i'th coefficient of the summed DCT block, then, if there are tion of watermarks after the watermarked data is subjected 

M blocks in the image, Ai/M can be added to each individual to predefined scale changes, without modification to the 

block, or block 1 can have Ai added to it and the remaining watermark detector. 

M-l blocks left unaltered, ignoring for the moment issues of Another object of the invention is the provision of a 

image fidelity. Because of this one to many mapping, it is 55 watermark detection method that is computationally inex- 

possible to alter the insertion algorithm without changing the pensive in either the MPEG or decompressed domains, 

decoder. This is a very important characteristic, since in A stiU othef object of the inveQtion ^ me ^ of a 

some watermarking applications there may be many hard- watermarking method that withstands attacks without 

ware decoders thai are deployed, such that changing the hayi tQ ch a detector 

decoder is impractical. However, improvements to the inser- 60 „ , , .„ . , 

tion algorithm can still result in improved detection using u Further and * nl } other ob J e * s °f * e mention will 

the approach described herein. become more clearl y a PP arent when the following descrip- 

A third advantage of watermarking the sum of the DCT tion * read " <™j u » with the accompanying drawings, 

blocks is that watermark signals extracted from these sums BRIEF DESCRIPTION OF THE DRAWINGS 

have small variances, compared with the amount that they 65 

may be changed without causing fidelity problems. This FIG. 1 is a general data flow diagram of a method of 

means that, in many cases, it is possible to change an image inserting a watermark into media data; 
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FIG. 2 is a schematic diagram of an MPEG-2 encoder; 

FIG. 3 is a schematic diagram of a modified MPEG-2 
encoder for reducing degradation of a watermark in water- 
marked data; 

FIG. 4 is a schematic diagram of an alternative modified 
MPEG-2 encoder for reducing degradation of a watermark 
in watermarked data; 

FIG. 5 is a flow chart of the process performed in the 
watermark correction device in FIGS. 3 and 4; 

FIG. 6 is a flow chart of the process performed in step 506 
of FIG. 5; 

FIG. 7 is a flow chart of an alternative process performed 
in step 506 of FIG. 5; 

FIG. 8 is a flow diagram of a method of extracting a 
watermark from MPEG media data; 

FIG. 9 is a flow diagram of a method of extracting a 
watermark from uncompressed media; 

FIG. 10 is a flow diagram of a method of detecting a 
watermark from MPEG data, with registration; and 

FIG. 11 is a flow diagram of a method of detecting a 
watermark from uncompressed image data, with registra- 
tion. 

DETAILED DESCRIPTION 

As used in the following description the terms image and 
image data will be understood to be equally applicable to 
video, image and multimedia data. The term "watermark" 
will be understood to include embedded data, symbols, 
images, instructions or any other identifying information. 

In order to better understand the present invention, first a 
review of the basic watermarking method will be presented 
followed by additional descriptions of the improvements 
comprising the present invention. 

First, we define some notations. Let a watermark to be 
embedded into an image be an N dimensional vector, 
denoted by W[l, . . . ,N]. In the following text, the notation 
W[l, . . . , N] is used in the same manner as W[k] (k-1, . . . , 
N). Let V[l, . . . ,N] denote a vector value extracted from an 
image, where the element V[k] corresponds to W[k]. 
Specifically, the value V[k] is a weighted sum of DCT 
coefficients given by 

V[*]-£'_A(3]f_A[l]+/>_A[2]f_A(2}f. . . +D_fin_k]F_lc[rt_k\ 

where D_k[i](i«l, . . . , n_k) indicate members of the set 
of DCT coefficients used for calculating V[k], n_k indicates 
the number of members, and F_k[i](i=l, . . . , n_k) are 
weighting coefficients related to a filter processing. The 
concept of F_Jc is that the DCT coefficients are weighted 
according to how much noise might be expected in each 
coefficient. To calculate V[l, . . . ,N], nxn DCT coefficients 
are first calculated over a whole image. Then the coefficients 
are classified into N sets, each of which is related to each 
element of V[l, , . . , N]. The rule of classifying DCT 
coefficients is predetermined, and the same rule is used in 
both inserting and detecting a watermark. 

Before inserting a watermark into an image, a detection 
algorithm is applied to the image to find a watermark that is 
already present in the image. If the image does not contain 
a watermark, the extracted values, V[l, . . . , N] will be 
normally distributed random numbers that do not correlate 
any watemrark W[l, . . . , N]. A watermark W[l, . . . , N] is 
inserted into an image by changing each of D_Jc[l, . . . , 
n__k] (k-1, . . . , N) slightly in order to make the extracted 
value V[l, . . . , N] highly correlate the watermark W[l, . . . , 
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N]. Let the target value of V[l, . . . , N] be denoted by 
o)[l, . . . , N], that is, the values V[l, . . . ,N] are changed to 
od[1, . . . , N] by inserting the watermark W[l, . . . , N]. The 
target values u)[l, . . . , N] have high correlation with the 
watermark W[l, . . . , N] and they are determined as will be 
described below. 

After the target values co[l, . . . , N] are determined, the 
difference o>[k]-V[k] is distributed among the DCT coeffi- 
cients D„k[l, . . . , n_k]. Then a watermark is inserted by 
adding the allocated difference value to the corresponding 
DCT coefficients D_k[l, . . . , n-k]. This change of DCT 
coefficients must be done in such a manner so as not to 
change the appearance of the image. 

In distributing the difference oo[k]-V[k], a characteristic 
of the human visual process is taken into account. The 
amount of change that does not cause a visible change in the 
image is different for each DCT coefficient D__k[i]. This 
amount depends on the human visual process which can be 
approximately simulated with a computational model. The 
amount of change is referred to as "slack". The slack is 
calculated for each DCT coefficient and it is used in distrib- 
uting the difference u)[k]-V[k] among the DCT coefficients. 
Next we describe how to calculate the values of the slack 
using a model of the human visual process. 

The preferred computational model of human visual sen- 
sitivity that is used in the present invention is found in an 
article by Andrew B. Watson, entitled "DCT Quantization 
Matrices Usually Optimized for Indirect Images" in SPIE, 
vol. 1913 (1993), pp. 202-216. This model was applied to 
watermarking in an article by Christine I. Podilchuk and 
Wenjun Zeng entitled "Digital Image Watermarking Using 
Visual Models", Proc. of EI'97, vol. 3016, Feb. 9-14, 1997. 
The current invention differs from that of Podilchuk and 
Zeng in (i) not requiring the original unwatermarked image 
at the decoder and (ii) not extracting the watermark from the 
individual 8x8 blocks, but from the sum of a set of 8x8 
blocks. Other computational models are also usable. 

For each element of the image's block DCT, d[ij], this 
model computes a value called the element's "slack", S[ij], 
which indicates how much a particular d[ij] value may be 
altered before such an alteration becomes visible. The value 
is computed in three steps. The first step models the contrast 
masking phenomenon of the human visual system and 
models the visual sensitivity at different frequencies and 
handles the difference between visual sensitivity to changes 
in different frequencies. The second step models the lumi- 
nance masking phenomenon of the human visual system and 
handles the fact that the visual system is more sensitive to 
changes in dark regions than to changes in bright regions. 
The third step handles the fact that the sensitivity to changes 
depends in part on the percentage that the frequency is 
changing (i.e. a DCT term with a small value in it may only 
change a little, while one with a larger value may change 
more). 

The perceptual model makes use of a matrix of values that 
indicate the relative sensitivity of the human visual system 
to the different terms of a spatial 8x8 DCT. The formulae for 
computing this matrix are available in an article by Albert J. 
Ahumada Jr. and Heidi A. Peterson entitled "Luminance- 
Model-Based DCT Quantization for Color Image 
Compression", in SPIE, vol. 1666, (1992) pp. 365-374. 

After computing the slacks for all the 8x8 DCT's in the 
image, a slack can be assigned to each D_k[l, . . . ,n_k]. 
Call these slacks S_k[l, . . . ,n_Jc]. It is now possible to 
distribute the changes in the V's over all the D_Jc*s with 
minimal visual impact. This is done according to the fol- 
lowing formula: 
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where D'_k[l, . . . , n_k] are the modified 8x8 DCT 
coefficients, and a)[k] and V[k] are the k'th elements of o> 
and V, respectively. The effect of this formula is to distribute 
the desired change in a given element of the watermark 
vector (co[k]-V[k]) over all the DCT coefficients that are 
summed to produce that element, proportionately according 
to those DCT coefficients' slacks. To illustrate, consider two 
simple examples: 1) If all the slacks are 0 except for slack 
S__k[m], then the sum of all the values S_k[j]F„k|j] is 
equal to S_k[m]F_k[m], and only D_k[m] is changed. It is 
changed by the full value of (co[k]-V[k]). 2) If all the slacks 
are equal and all the coefficients F_Jc[i]~l, then each D_k[i] 
is changed by the same amount. 

After making these changes, convert all the 8x8 DCT's 
back into the spatial domain, and the result is a watermarked 
image. It is easy to show that the sum of all the D_Jc's for 
a given k will equal to[k]. The process of making this is 
referred to as "inserting Omega into the image". The water- 
mark extracted from the resulting image, if the image has not 
been attacked, will be exactly co, not o> plus noise. 

There are two important issues remaining to be discussed. 
First, how to decide on to, and, second, how to make the 
watermark robust. 

Previously, the equivalent of to was computed as: 

where a is a small constant, and W is a zero-mean water- 
mark signal. It is possible to use the same formula here, but 
it is too limiting to result in the strongest possible watermark 
using the present invention. In practicing the invention, it is 
often possible to insert an co that has perfect correlation with 
the watermark, W, without causing any visible change in the 
image. The following formula is used: 

o>-mean(V)+p*(V-meart( V))+a* W 

This result is a weighted sum of the watermark signal and 
the original, noise (image) signal. If p is set to 0, the result 
is an co that perfectly correlates with W. 

The signal to noise ratio for an unattacked image will be: 

SNR-a*std(W)ip+std(V) 

where std(X) is the standard deviation of X. 

There are many ways to choose a and p based on 
optimizations to maximize different criteria such as fidelity 
or robustness. 

At this point there is a complete method of inserting 
watermarks. The method contains explicit modeling of 
human vision, but it does not contain any explicit method of 
making the watermark robust. In fact, the method as 
described so far will try to put as much of the watermark as 
possible into the high frequencies since these frequencies 
have the largest slack, but this is a poor thing to do from the 
point of view of robustness. 

To make the watermark robust against a given set of 
attacks or signal degradations, it is first necessary to consider 
how those attacks affect the various terms of the 8x8 DCT's 
in the image. Then, terms that are affected by attacks or 
signal degradations in similar ways are grouped together, 
and watermarked as if the group of terms were a separate 
image. 
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The following is a simple example. Suppose there is only 
concern about two possible attacks: cropping 24 columns of 
pixels from the left side of the image, or cropping 24 
columns of pixels from the right side of the image. This 

5 results in three groups of DCT terms: those that come from 
the 3 left-most columns of 8x8 DCT's blocks, those that 
come from the 3 right-most columns, and those that come 
from the rest of the image. All the terms in each of these 
groups either survives or is destroyed by any given attack 

30 together. If each group is watermarked as though it were a 
separate image, then the watermark from at least one group 
will generally survive attack (assuming that the 24-column 
cropping attacks are the only attacks possible), and the 
watermark that is extracted will consist of the correct 

15 watermark, from that group, plus some noisy watermarks 
from groups that were damaged by the attack. 

A more interesting example is low-pass and high-pass 
filtering attacks. It is possible to group all the low frequen- 
cies together into one group, and all the high frequencies 

20 into one or more other groups. If the predetermined rule for 
classifying DCT coefficients into the N sets is designed in 
such a way that each set has coefficients of many different 
frequencies, then the complete watermark can be inserted 
into each group. Then, if the high frequencies are removed, 

25 the watermark will still be detectable in the low frequencies, 
and vice-versa. 

The more groups the terms are divided into, the more 
robust will be the watermark. There is a cost because it will 
become increasingly more difficult to distribute the changes 

30 without causing visible distortion. 

It is important to note that the best balance can be 
achieved after the detectors are in wide use. It is possible to 
modify the insertion algorithm to make watermarks robust 
against a wide variety of attacks without having to change 

35 detection at all. 

Presently only three groups are used. One group collects 
together most of the low frequencies. Each of the other two 
groups represents one higher frequency. 

In the preferred method, a watermark is not placed in any 

40 of the higher frequencies. The reason is not that they are 
susceptible to attack (that is handled by the design of the 
Filter [ ] used in detection). Rather, it is because watermark- 
ing the higher frequencies causes MPEG compression rates 
to go down substantially. An alternative solution might be to 

45 add other groups that contain higher frequencies. 

In FIG. 1, there is shown a flow diagram of the watermark 
insertion method. The digital image is divided into a col- 
lection of nxn blocks, preferably 8x8 blocks, in step 10. The 
discrete cosine transform (DCT) of each block is computed 

50 in a known manner in step 12. The DCTs are separated into 
groups that respond to different attacks in the same manner 
14. A first group G is selected in step 16. 

Next, extract a watermark V, using only the terms in the 
group G in step 18. Determine a new signal (target value) co 

55 selected such that co is similar to V but is highly correlated 
with watermark W in step 20. 

Add fractions of co-V to terms in G according to percep- 
tual slack in step 22. Decide whether group G is the last 
group in step 24. If not, select next group G in step 26, and 

60 extract watermark V using only terms of next group G in 
step 18 and continue procedures until the last group G is 
found in step 24. Next, compute the inverse DCTs of the 
blocks in step 28 resulting in a watermarked image. 
Some alternative steps in the insertion method are pos- 
es sible. For example, the distribution of the difference between 
to and V over DCT terms can be done stochastically to help 
deter tampering and reduce susceptibility to tampering. 



09/08/2004, EAST Version: 1.4.1 



6,154,571 

9 10 

Also, the groups of DCT terms for robustness purposes coefficients for the decoded images which are expected to be 

could be performed dynamically. For example, the insertion generated in a decoder. These DCT coefficients are inputted 

program could simulate various attacks on the image and into a watermark correction device 62. The watermark 

determine the effect on the values of the DCT term. Then, correction device 62 outputs watermark correction signals, 

the program would cause appropriate allocation of the terms 5 At adder 64, the watermark correction signals from device 

into the groups similarly affected. 62 are added to the quantization values from quantizer 36. 

The inserter can be designed with a user-interface that The output of adder 64 is used as the inputs to variable 

allows the user to set two parameters: (1) the maximum length encoder 38 and inverse quantizer 40. 

perceptual difference between the original image and the Next, we describe the process performed in the watermark 

watermarked image (J) and (2) the maximum allowable 10 correction device 62. Let us introduce several new notations 

probability of missed detection after any of the predefined to explain the process. Let Dq_k[i] be the quantization 

set of attacks. The algorithm would then insert watermarks value corresponding to D_Jc[i], that is, the quantization 

into a large number of images automatically, according to value of the i-th member of the k-th set for calculating the 

the allowable perceptual change J and checking each one value V. Let Q_k[i] be the quantization step size used in 

against simulations of the attacks. If an image fails to meet 15 obtaining Dq_k[i]. Let Dr__k[i] be the output value of adder 

the specified robustness constraint (maximum allowable 60 that is obtained by adding the inverse quantization value 

probability of missed detection), then the user would be of Dq_k[i] calculated in inverse quantizer 58 to the corre- 

notified so that a manual decision can be made to compen- sponding DCT coefficient outputted from DCT calculator 

sate or trade-off image fidelity for robustness. 56. Let Vr[l, . . , ,N] be the value extracted from the output 

In addition, the distribution of the difference between to 20 values of adder 60, Dr__k[i], in the same manner that the 
and V over DCT terms can be modified to explicitly com- value V[l, . . . , N] is calculated in inserting a watermark. We 
pensate for MPEG quantization. Using the above watermark assume that the target value co[l, . , . , N] is also available 
insertion method there may result a degraded watermark in in the watermark correction device 62. FIG. 5 shows that the 
the watermarked data. In order to enhance the watermark in flow chart describing the process performed in the water- 
the watermarked data after MPEG compression several 25 mark correction device 62. First the index k of the water- 
techniques are possible. mark element is set to l(Step 500). Next the value Vr[k] is 

FIG, 2 is a schematic diagram of a typical MPEG-2 calculated (Step 502) by 
encoder. FIG. 2 depicts elements which are indispensable to 

execute an MPEG-2 encoding of P pictures, or to perform a WfrF-JVIPr-Ai}*- • • F_k{n_k]pr_k{n_k\ 

combined interframe prediction and DCT coding. Input 30 whefe me wei htin coeffici ents F_k[l, . . , ,n_k] are the 

images are provided as one input to subtractor 30 The other same yahlc ^ tQOSe used in calculati y[k]. Then the 

mput to subtracter 30 is predicted image generated in frame absolute value of ^ difference betweeQ tne value Vr[k] and 

memory 32. The predicted images are subtracted from the ^ yame ^ and ^ si of me difference are 

mput images at subtractor 30. A discrete cosine transform computed m step 504 by the following equations: 

(DCT) is performed at DCT calculator 34 on the output 35 v Y J & M 

signal from subtractor 30. The DCT coefficients are quan- Dif-\Vr{k}-<i>[k]\ 
tized in quantizer36. The outputs of the quantizer 36 are sent 

to a variable length encoder 38 where Huffman encoding is s-siGNW}-a[kj), 

performed. The quantized DCT coefficients outputted from where 

the quantizer 36 are also sent to an inverse quantizer 40 40 

where they are de-quantized. Inverse DCT of the , { (;ca0) 

de-quantized DCT coefficients is performed in the inverse SignU) s \ x {x < 0) 
DCT calculator 42. The results are added at adder 44 to the 
predicted image outputted from the frame memory 32, and 

then an image which is expected to be the same as that 45 This value Dif corresponds to the distortion of the water- 
acquired in a decoder is reconstructed. The reconstructed mark inserted at the adder 52 generated in the quantization 
image is called "a locally decoded image." This locally process. 

decoded image is stored in the frame memory 32 to produce On the basis of the absolute value Dif and the sign s 

the predicted images. watermark correction signals are generated in step 506. The 

FIG, 3 is a schematic diagram of a modified MPEG-2 50 process in Step 506 is described below. After calculating the 

encoder for reducing degradation of a watermark in water- watermark correction signals for the DCT coefficients 

marked data. Before MPEG-2 encoding, DCT is performed related to the k-th element of the watermark, index k is 

on a input image at the DCT calculator 50 and watermark compared with N in step 508. If k>N. then the process is 

signals are added to the DCT coefficients at adder 52. The finished. If k^N, the value k is increased by one in step 510, 

output DCT coefficients including watermark information is 55 and the process goes back to step 502. The watermark 

subject to inverse DCT in the inverse DCT calculator 54. correction process is thus performed, and the obtained 

The output of inverse DCT calculator 54 are images with a watermark correction signals are finally outputted to adder 

watermark. These watermarked images are sent to MPEG-2 64. 

encoder and MPEG-2 encoding is performed as described Next, we describe the process performed in step 506 in 

above. In addition, in this embodiment, watermark informa- 60 FIG. 5, using the flow chart in FIG. 6. In step 506, the 

tion is modified in order to be suited to MPEG-2 compres- watermark correction signals for the DCT related to the k-th 

sion. DCT coefficients for the predicted images are calcu- element of the watermark are generated. The array ADq_k 

lated in DCT calculator 56. The quantization values [i](i-l, . . . n_Jc) are first all set to zero in step 520. Next, 

outputted from quantizer 36 are de -quantized in inverse the value j is set to one in step 522. Then the index of the 

quantizer 58. The results of inverse quanitization are added 65 DCT coefficients i is found through a permuting function 

at adder 60 to the DCT coefficients outputted from DCT p(j) in step 523. The function p(j) returns the j-th value of a 

calculator 56. The results of addition correspond to the DCT permutation obtained after the integers 1 to n_k are per- 
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muted. The simplest example is p(j)-j- Next, a value -s is 
stored in ADq_k[i], and the value Dif is decreased by 
Q_Jc[i]F_k[i] in step 524. This indicates that the quantiza- 
tion value Dq_k[i] is changed by -s by adding ADq_k[i] to 
Dq_k[i] at adder 64. The value s is 1 or -1, so the change 5 
in the quantization value is one. In other words, 
de-quantized value obtained in a inverse quantizer in a 
decoder is changed by -Q_k[i], one step size. The value Dif 
after the update is identical to the absolute value of the 
difference between co[k] and Vr[k] calculated with the cor- 
rected quantization values Dq_k[i]+ADq_k[i](i=l, . . . 
n_k). After step 524, the values Dif and zero, and index j 
and n_k are compared in step 526. If Dif<0 or j>n_k, then 
this subroutine is finished. If the condition is not satisfied, 15 
the index j is increased by certain amount Aj in step 528, 
then the process returns to step 523. 

Instead of the process shown in FIG. 6, an alternative 
process shown in FIG. 7 can be used as the process of step 
506. In the process shown in FIG. 7, a step 530, checking 20 
whether the quantization value Dq__k[i] equals zero or not, 
is added between step 523 and step 524. In this case, step 
524 is performed only if the quantization value Dq__k[i] is 
a non-zero value. This allows a reduction in the increase of 
the number of bits caused by correcting watermark 25 
information, because changing a quantization value from 
zero to non-zero value generally results in a large increase 
in the number of bits. 

FIG. 4 is a schematic diagram of an alternative embodi- 30 
ment of a modified MPEG-2 encoder for reducing degrada- 
tion of a watermark in watermarked data. In this 
embodiment, the basic concept is the same as the described 
in connection with FIG. 3. The differences lie in the fact that 
subtraction of the predicted images from the input original 35 
images is performed in the DCT domain not in the spatial 
domain. For the predicted image outputted from the frame 
memory 32, DCT is performed in DCT calculator 70, and 
the results are subtracted from the watermarked DCT coef- 
ficients at subtractor 72. The results of subtraction are sent 40 
to the quantizer 36 and then the watermark correction is 
performed in the same manner as shown in FIG. 3. The 
results outputted from subtractor 72 are the same as the 
results outputted from the DCT calculator 34 in FIG. 3 
because of the linearity of DCT. Therefore, the results 45 
obtained in the processes followed by quantization in quan- 
tizer 36 are the same as those in FIG. 3. This embodiment 
results in a reduction in the number of DCT calculations. 

The detection procedure to detect a watermark in an 
image will now be described. 50 

If MPEG video is the input image data format, the 
following detection process determines whether watermark 
W is present, where W[l, . . . , N]»the watermark being 
tested for. 55 

Decode the Huffman code, but do not compute the inverse 
DCT's, so that, for each frame (at least, each I-frame), there 
is an array of 8x8 DCTs. 

Next perform the same summation of DCT coefficients 
that was performed during watermark insertion to obtain the 60 
vector V. Compute the correlation coefficient C, between V 
and the watermark being tested for, W: 

V = V -V 

65 

W' = W -W 



12 

-continued 

W'-V 

c= 

V(W'- w")(v-v") 

Finally, convert C into a normalized Fisher Z statistic: 



V/V-3 1 -*-C 

Z= log 

2 1-C 

where N is the length of the watermark. 

The Z value indicates whether the watermark is present. 
A preferred threshold for Z is 4 (i.e. Z^4 means the 
watermark is present), but other values may be used depend- 
ing on the desired probabilities of false alarms and missed 
detections. 

FIG. 8 is a flow diagram of the detection method for 
MPEG video input described above. The input MPEG video 
is subject to a Huflinan decoder and partial parser 80 where 
the output is a set of DCT for nxn, preferably 8x8, blocks of 
the video input. 

The nxn DCTs are provided to watermark accumulator 
82. Accumulator 82 has memory whose length is the water- 
mark length N. DCT coefficients from the Huffman decoder 
and partial parser 80 are classified according to a predeter- 
mined rule and summed for extracting a watermark as 
mentioned before, and the results are accumulated in the 
memory. The extracted watermark is proved to comparator 
84 where it is compared with possible watermarks in the 
image by calculating correlation coefficients between the 
extracted watermark and the possible watermarks as men- 
tioned before. The possible watermarks are the universe of 
the watermarks accumulators and comparator are found, for 
instance, in U.S. patent application Sen No. 08/746,022. 

The output of comparator 84 is the likelihood (normalized 
Fisher Z statistic) of the detected watermark being each of 
the possible watermarks. The most likely watermark is 
determined and is deemed the watermark in the image, or, if 
the detector does not exceed a predetermined threshold, then 
no watermark is present. 

Alternatively, if the incoming input data comprises an 
uncompressed image, an embedded watermark can be 
detected by applying the method above to DCT coefficients 
obtained by performing 8x8 DCT for the whole image. In 
this case, DCT have to be performed for each 8x8 block, but 
a skillfully designed rule for classifying DCT coefficients 
into N sets enables us to avoid performing DCT many times. 
Before explaining the method to reduce DCT calculation, let 
us define some notations. 

Let h_m(i, j) (i«l . . . 8 ... 8. m«0, . . . ,M-1) be 
a set of functions that map frequency indices of 8x8 DCT 
coefficients (i, j) onto the indices k of the element of a 
watermark, and M indicates the number of functions. So if 
k«h_m(i, j), then a DCT coefficient whose index is (i, j) is 
classified into the k-th set of DCT coefficients for calculating 
the value V[k]. We prepare M different functions h_jn(i,j) 
(m-0, . . . M-l). Which function is selected for a certain 8x8 
block depends on the numbers r and c of the block where r 
and c indicate the row and column numbers of the block 
respectively. So the index of the functions, m, is first found 
according to the values r and c, then the index of the sets, k, 
is determined by h„m(ij) for each DCT coefficient. Using 
these functions h_m(i j) in classifying DCT coefficients is 
assumed in the remaining part of the detailed description. 

In this case, we can reduce the number of DCT calcula- 
tions in the following manner. First the sum of the blocks 
whose indices m are the same is computed for each 
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m«0, . . . , M-l. Let this summed block be denoted by 
VB_m[ij](m«0, . . . ,M-1). Then DCT is perfomred for the 
M summed blocks VB_m[ij](m«0,. . . , M-l). Finally, the 
DCT coefficients of the summed blocks are classified into N 
sets according to the value h_m(ij), and added together 5 
within each set to obtain V[l, . . . ,N]. The obtained results 
V[l, . . . N] are the same as V[l, . . . N] obtained with the 
method mentioned above because DCT is a linear transform, 
that is, the sum of DCT blocks equals the result of DCT for 
the sum of the blocks. If M is much less than the total 10 
number of blocks in an image, the number of DCT calcu- 
lations is dramatically reduced. This method thus allows us 
to extract watermarks with small calculation cost. 

FIG. 9 shows a flow diagram of the detection method for 
uncompressed video input data as described above. 35 

The uncompressed video image data is provided to nxn 
accumulators, preferably 8x8 accumulators 90. The memory 
requirement is n 2 times the number of the function h_m(ij). 
For each index m. the blocks with the same index m are 
summed and the resultant M summed blocks are accumu- 20 
lated in the memory. 

The output is the summed signal of each of the nxn 
blocks. The output is subject to a DCT transform 92. The 
number of transformations is proportional to the number of 
the functions h_m(ij). The result is a group of nxn DCTs 25 
which is classified into N sets according to the functions 
h_m(i j) and summed for extracting a watermark as men- 
tioned before. The obtained watermark is provided to water- 
mark accumulator 94 and accumulated. The memory 
requirement for accumulator 94 is proportional to the water- 30 
mark length. The extracted watermark is provided as input 
to comparator 96. The other inputs to comparator 96 are the 
possible watermarks that may have been inserted into the 
input image data. The comparator computes a likelihood 
(normalized Fisher Z statistic) of each possible watermark 35 
having been inserted into the image data. The most likely 
watermark is determined and is deemed the watermark in the 
image. 

A limitation of block based DCT methods is their sensi- 
tivity to spatial shifts of the image. For example, if the image 40 
is shifted two pixels to the right, then the DCT coefficients 
change significantly, so that the watermark cannot be 
detected. Furthermore, general distortions, such as scaling 
and rotation, also make the watermark undetectable. 

To solve these problems, the above insertion and extrac- 45 
tion methods may be modified in two ways. The first 
possible modification is to insert multiple watermarks 
designed to survive predefined distortions of the video. The 
second modification is to arrange that translations can be 
compensated for without performing the summation more 50 
than once. Optionally, this second modification may be 
further modified to insert registration patterns, one for each 
of the multiple watermarks, which can be used by a modified 
watermark detector to compensate for arbitrary translations 
of the video. 55 

When detecting watermarks inserted using the above 
method, it is necessary to divide the image into the same grid 
of nxn blocks as was used during insertion. If the image has 
been translated since watermark insertion, then determining 
the correct grid becomes difficult. In many applications, this 60 
is a serious problem, since certain, specific transformations 
can be expected. For example, it can be expected that video 
on a DVD disk might be modified to fit on a standard 
television screen by conversion to either "panscan" or 
"letterbox" mode. In "panscan" mode, the horizontal image 65 
resolution is increased, and the image is cropped at a 
predetermined offset, so that the resulting image will be 
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correct when viewed on a 3x4-aspect-ratio television screen. 
In "letterbox" mode, the image is scaled vertically, and black 
is added at the top and bottom, so that the whole image will 
fit correctly on a 3x4-aspect-ratio screen. Since these two 
geometric transformations are more likely than any other, it 
is reasonable to prepare for them specifically. 

The problem of the predetermined scaling or transforming 
of watermarked video is solved in the present invention by 
inserting an additional watermark for each of the likely 
transformations. Each of these watermarks is designed so 
that, when the image has undergone the corresponding 
known transformation, the grid of nxn blocks used during 
insertion will align with a predetermined grid used during 
detection. Thus, if the image has undergone no 
transformation, then the detection grid will align with the 
normal mode watermark, and the normal mode watermark 
will be detected. If the image has undergone "panscan" 
transformation, then the same detection grid will align with 
the "panscan" watermark, and the "panscan" watermark will 
be detected, and so forth for "letterbox" scan or any other 
predefined transformation. 

The procedure for inserting a watermark that is to be 
detected after a specific transformation comprises the fol- 
lowing steps: 

1. Make a copy, l r , of the image being watermarked, I, 
and apply the transformation to be compensated for the 
image. For example, \ T might be a copy of I that has 
been transformed into "letterbox" mode by vertical 
shrinking of I. 

2. Create a watermarked version of the transformed 
image, I r \ according to the general watermarking 
method described above. 

3. Let Wj^Ij-'-Ij- be the spatial pattern that was added to 
l r when it was watermarked. 

4. Perform the inverse transformation on W r to yield the 
corresponding watermark pattern, W, for the untrans- 
formed image. For example, if the transformation to be 
compensated for was "letterbox" mode, then W would 
be obtained by vertically expanding W T . 

5. Let I'al+W be the image with a watermark added for 
the given transformation. 

When the transformation is applied to T, the result will be 
approximately l T \ and the watermark will be detected by the 
same procedure designed to detect a normal watermark. 

This process can only be used for a small number of 
transformations, as each additional watermark causes addi- 
tional degradation of the image, and reduces the detect abil- 
ity of other watermarks in the image. However, tests have 
shown that three watermarks — two for transformed images 
and one for the untransformed image — result in acceptable 
fidelity and good detectability. Alternatively, for video, each 
watermark can be inserted in a time-multiplexed manner. 

In cases where the transformations that an image will 
undergo are not predefined or predetermined, or where there 
are too many probable transformations to allow for the 
insertion of a separate watermark for each transformation, 
the above described method of compensating for transfor- 
mations is not optimal. Thus, the present invention includes 
an additional improvement, which compensates for arbitrary 
translation of the image between the times of watermark 
insertion and watermark detection. 

Arbitrary translation is compensated for in two ways. One 
way is by translations by even multiples of 8 pixels in the x 
or y directions when 8x8 blocks are used. This can be easily 
compensated for if the following restrictions are imposed on 
the relationship between an index of h_m(i j), m, and the 
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row and column numbers of blocks, r and c. We determine can thus insert a registration pattern using a method similar 

the index m according to to that inserting a watermark described above, considering 

the DCT coefficients of a registration pattern as a watermark 

m-f(r,c)mod M, W and considering the sum of all 8x8 DCT blocks as a value 

5 V 

where f(r,c) is a linear function of r and c, and f(0,l) and Insertion is performed by converting the registration 

f(l,0) are integers. In addition, let h_m(ij) be a function pattern into the DCT domain, and then using the basic 

expressed as insertion algorithm with different D^s and F^s. Specifically, 

each AC term of the registration pattern's DCT is considered 

A^(w>(*_0(y)+««»)«flrf one element of the watermark, W[k]. The set of DCT terms 

, . 4 ™ , . .. , . that are summed together to extract this element, D*, is 

where a is an integer. These restrictions are assumed in the , . * j- c n *u o o 

• • ^ i ■ a jj . 4 . T iL . simply the set of corresponding terms of all the 8x8 DCTs 

remaining part of the detailed description. In this case, the • .*f • AmL r * 1 n * *l ji- 

,. ri ° r . - * j * *• in the image. All the F*s are set to 1. Using these D*s and F*s, 

shut compensation is performed in the basic detection ... _r . -.u • . • . i 

, * r , c . ± jx the insertion algonthm inserts a registration pattern along 

algorithm by computing the correlation of the extracted , & . ™ f .... . . f 

. , *V, ? i i_-Ji r *t. * i i_ • . with each watermark. The watermarks are still mserted with 

watermark with all cychcal shifts of the watermark being is ^ ^ . ^ ~ ^ p 

tested for. This is because the values VB m[ij] (m=0, . . . , ~ P , . • A • * r 

w «\ i_* • j r * j « During detection, registration is performed as follows. 

M-l) obtained from a watermarked image shifted by a „ , J 9 j * • j L e L i i j 

u . , r o * i ■ .i. t_ • .i j/ .i_ *• i Each of a predetermined number of blocks is summed 

multiple of 8 pixels in the horizontal and/or the vertical . iU t / . , n 0 0 , . . . 

,. 5 >j . . . « , trri r . rt together to form a single nxn, typically 8x8, block. We have 

directions are identical to the values VB__m[i j ](m=0, ... , * , , . & . . J \ t ' , 

w i\ i_. • jr u-jv j • * lL ' iL arbitranly used 64 blocks in our tests. This block contains a 

M-l) obtained from the non-shifted image except that the 20 . A 4 . 4i i_i.i_l.u- -n 

. ' „ u-f* j a i. *u * ♦ -i registration pattern placed there by the insertion process. To 

indices m are cyclically shifted. As a result, the extracted , f . , . r _ . j _. » _ i _ • « . . 

i itt* ktt • j c , i _i * determine the honzontal and vertical translation of the 

value V[l,. . . ,N] obtained from a watermarked image c ... . - , . ... 

u-c* j u n- i ro j ** 1 * *u t \/n frame, a correlation process is performed in the spatial 

shifted by a multiple of 8 are identical to the value Vfl, . . . , , .' . , _ . \, ~ _ r „. _ „ . 

*n i__ • _i r _i_ t_ r. j ■ _ .l . domain to determine these oifsets. Sixty-four correlations 

N] obtained from the non-shifted image except that the c , c __ r ___ o i_ • _ i _io _• i 

i , . »pi i ,. n , , c . . r , iU „_ are performed for each of the 8 horizontal and 8 vertical 

elements V[k] are cyclically shifted. For nxn gnd format, the 25 J , , 17U 0 0 . , . ... , 

... L . J t 3 c r ... f - . V. motions that are possible. When the 8x8 patch is shifted 

same thing is true of a shift of a multiple of n in the ... , . _ n r . r , ..^ , 

horizontal and/or the vertical directions. "erformeT 0 " * ^ " 

For shifts of less than 8 pixels in x and/or y directions, an ^ e ' ... ... . . r. r . 

, .. . u j r 11 iv* -u-i**' We now describe how to determine the shift of the grid, 

exhaustive search can be performed of all 64 possibilities i__.ru- • . t r^. - x 

and the maximum Z value taken from the set of 64xM-Z 30 In ' h ^ fo " owin8 ^ tho , d ' we integer values f(0,l) 

i < » X • .< u o i * t and f(l,0) are relatively pnme to M. 

values, where M is the number of 8x8 accumulators. In our ~. V Y . . , f f.. . •_ -x ■ 

4 , . .u/r^TT-i***** * First, the blocks for which the same function h m(ij) is 

tests M was chosen to be 64. The factor M is necessary to , . , . nk ™ «= ■ . j j j . .u • 

. £ . . f . . . * , . . c used in classifying DCT coefficients are added together in 

account for the cyclic shifts that are introduced by shifts of 4 . .• . _ • r .* , ™ 

a multi le of 8 ixels spatial domain for generating a summed block VB__m 

p . . . XM o o fiJl for each m. The sum of all nxn blocks denoted by 

The exhaustive search requires shifting the M 8x8 accu- 35 . , . 7 

mulator array in the spatial domain and then performing the ^ ^ ^ 

DCT of each of the M 8x8 blocks. This is performed 64 

times, once for each of the possible shifts. Thus it is AB ^ j^-y VB m [/ /] (/=i n j=\ n) 

necessary to perform 64xM 8x8 DCTs. If this computation ' ^ ~ ' 1 1 

is too expensive in terms of time or memory an alternative 40 
method can be used, as described below. 

The second way, which compensates for translations of the correlation coefficient between AB[ij] and a 

non-even multiples of n pixels, uses a pattern (referred to as registration pattern, denoted by R[ij] is computed. After 

a "registration pattern") which can be inserted at the time of calculation of the correlation coefficient, the values AB[ij] 

watermark insertion. By finding the location where the 45 are cyclically shifted by one column in the horizontal 

registration signal best matches a predefined signal, a detec- direction, and the correlation coefficient between AB[i j] and 

tor can determine how much to shift the data before extract- R [* j] is calculated in the same way. The same operatioas are 

ing the watermark. This shifting must be done in the spatial repeated for each shift. After shifting n times, AB[ij] 

domain, but can be done with accumulators, so conversions becomes identical to AB[i,j] before any shift is done. Then, 

of whole images are avoided. 50 lhe values ABPol are cyclically shifted by one row in the 

Moreover, the 64xM 8x8 DCTs are unnecessary. Instead, vertical direction, and calculation of a correlation coefficient 

the correct registration is determined in the spatial domain and shift b y one column in the horizontal direction are 

and then compensated for by shifting the pixels in the repeated. In this way, we can calculate correlation coeffi- 

accumulator arrays. The M 8x8 accumulators are only then cients for aU n possible shifts. At the same time, we search 

transformed into the DCT domain and the watermark extrac- 55 lhe shift value (ofl&et), denoted by (X,Y), which gives the 

tion is performed as described above. maximum correlation coefficient. 

The registration pattern is an 8x8 spatial pattern inserted MtCT the offiset has been determined, the M 

into the image in such a way that the sum of all 8x8 pixel summed blocks VB_m[io] are then shifted accordingly in 

blocks highly correlates with the pattern. Again, an nxn lhe SP 311 * 1 d °main. Next, we describe the method to com- 

spatial pattern is used when the video is nxn blocks. A 60 P ens ate for the shift value X in the horizontal direction. To 

registration pattern can be inserted by using the watermark do ^ the vaJues VB_m[ij](m-0, . . . ,M-1) are first copied 

insertion method described above. The sum of all 8x8 pixel on ™ nxnM arra y VBlp j] . . - ,n.j-l, . . . ,nM). copying 

blocks becomes highly correlated with a registration pattern the values VB_m[ij], the spatial relationships between 

if the DCT coefficients of the sum block and those of the blocks, that is, which blocks are adjoining a certain block are 

registration pattern are highly correlated to each other. In 65 considered. The function f(r,c) is linear, so 
addition, the DCT coefficients of the sum block equals the 

sum of all DCT blocks because of the linearity of DCT. We Ar,c+i)*j{r,c)+f{o,i). 
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I, the array VB1 filled with the values VB_m[ij] 10 potions r^m (ij) and summed for extracting a watermark. 



This indicates that the blocks whose index m equals 

are located next to the blocks in which m=mO. So, the values 
VB_ml are copied next to the values VB_mO in the array 
VB1. For this reason, for each I (1=0, . . . ,M-1), the values 
VB_m'[i, j](i=l, . . . ,nj=l, . . . , n, m'=IxF(0,l)mod M) are 
copied in the nxn square region of VB1 where the index of 
the left-top corner is (1, nl+l). After copying the values for 
all 

because f(0,l) and M are relatively prime. Next, the values 
in the array VB1 are cyclically shifted by X in the horizontal 
direction. Then the values in VB1 are returned to VB_jn in 
such a way that the value in the nxn region of VB1 where 
the index of the left-top corner is (1, nl+l) are substituted to 
VB_m'[ij](i=l, . . . ,n, j=l,. . . ,n, m'=lxf(0,l) mod M) for 
each 1(1=0, , . . ,M-1). The horizontal offset value X can thus 
be compensated for without shifting the whole image itself. 

Next, the shift value Y in the vertical direction is com- 
pensated for. To do so, the values VB_m[ij](m=0, . . . , 
M-l) are first copied on an nMxn array VB2 [ij] . . . , 
nM, j-1, . , . ,n). In copying the values VB_m[ij], the 
spatial relationship between blocks are considered. The 
function f(r,c) is linear, so 

f{r + l,c)=f{r,c)+f{l,0). 

This indicates that the blocks whose index m equals 

ml-(mO+/(l,0))mf?£/ M 

locate under the blocks in which m=m0. So, the values 
VB_ml are copied under the values VB_m0 in the array 
VB2. For this reason, for each 1(1=0, . . . ,M-1), the values 
VB_m'[ijXi-l, • - • >n, j=l, . . . , n, m'=lxf(0,l) mod M) are 
copied on the nxn square region of VBI where the index of 35 
the left-top corner is (nl+l, 1). After copying the values for 
all I, the array VB2 is filled with the values VB_m[i,j] 
because f(l,0) and M are relatively prime. Next, the values 
in the array VB2 are cyclically shifted by Y in the vertical 
direction. Then the values in VB2 are returned to VB_m in 40 
such a way that the value in the nxn region of VB2 where 
the index of the left-top corner is (nl+l, 1) are substituted in 
VB_m'[ij](i=l, . . . n, j-1, . . . ,n, m'=IxF(0,l) mod M) for 
each I (1=0,. . . ,M-1). The vertical offset value Y can thus 
be compensated without shifting the whole image itself. 

The offset of the n x n grid is compensated by the process 
mentioned above. These processes are performed in the 
registration process 108 as will be explained later. A shift of 
a multiple of n remains even after these processes, but this 
shift does not affect the watermark detection because the 
correlation coefficient between a watermark W and an 
extracted value V is calculated by shifting the watermark W 
cyclically as described above. After the registration process 
above is applied, the M blocks VB_m(m«0, . . . ,M-1) are 
transformed back to the DCT domain. 

With reference now to FIGS. 10 and 11, there are shown 
the basic detection algorithms modified to compensate for 
translational registration. In the case of MPEG video input 
(FIG. 10), 8x8 DCT blocks obtained from an MPEG video 
stream are first classified into M groups according to their 60 
indices no of the function h__m(ij), summed within the 
groups for generating M summed blocks, and the resultant 
summed blocks are accumulated in 8x8 accumulators 102. 
The M summed blocks in accumulators 102 must be con- 
verted into the spatial domain by performing an inverse DCT 65 
operation in inverse DCT converter 104, and accumulated in 
accumulators 106. Finding the offset value of the 8x8 grid 



and compensating for the offset is executed for the output 
from 8x8 accumulators 106 in registration 108 as described 
above. The registration data outputted from registration 
process 108 is accumulated in accumulators 110 and con- 
verted into the DCT domain in DCT converter 112 for 
watermark extraction by use of accumulators 114, water- 
mark extractor 116 and watermark decoder 118. In water- 
mark extractor 116, the DCT coefficients outputted from 
accumulator 114 are classified into N sets according to the 



The obtained watermark is provided to watermark decoder 
118, in which the processes executed in comparator 84 in 
FIG. 8 for finding a watermark corresponding to the 
extracted watermark. The watermark considered to have 
15 been inserted is outputted from the watermark decoder 118. 
In the case of uncompressed input data (FIG. 11), the input 
data is divided into 8x8 blocks and accumulated in accu- 
mulators 106 according to the indices of the functions 
h_m(ij), and registration 108 is performed before conver- 
20 sion into the DCT domain in DCT converter 112. The 
process continues as described above. 

While there has been described and illustrated methods of 
insertion and detection of watermarks in image data, it will 
be understood by those skilled in the art that variations and 
25 modifications are possible without deviating from the spirit 
and broad teachings of the present invention which shall be 
limited solely by the scope of the claims appended hereto. 
What is claimed is: 

1. A method of inserting a watermark signal into video 
30 images comprising the steps of: 

receiving input video images; 

performing discrete cosine transformation (DCT) of said 
input video images to obtain DCT values of said input 
video images; 

adding watermark signals to said DCT values to obtain 

DCT values with watermark; 
performing an inverse DCT on the DCT values with 

watermark for generating watermarked images; 
subtracting predicted images from said watermarked 

images for generating residual images; 
obtaining the DCT values of residual images and quan- 
tizing the DCT values of the residual images; 
inverse quantizing the quantized DCT values of the 

residual images; 
performing discrete cosine transformation of the pre- 
dicted images; 
summing the inverse quantized DCT values of the 
residual images and the DCT of the predicted images 
for generating DCT coefficients for decoded images; 
calculating correction signals from the DCT coefficients 
of the decoded images and adding the correction sig- 
nals to the quantized DCT values of the residual images 
for obtaining an output signal; 
inverse quantizing the output signal, inverse DCT the 
inverse quantized output signal and summing the 
resultant signal with the predicted images for generat- 
ing a summed signal; 
storing the summed signal in memory for generating the 

predicted images; and 
variable length encoding the output signal for providing a 
watermarked MPEG video signal of the input video 
images. 

2. A method of inserting a watermark signal into video 
images as set forth in claim 1, where said calculating step 
generates a negative value as said correction signal when the 
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extracted value exceeds the corresponding target value, and 
generates a positive value as said correction signal when the 
extracted value falls short of the corresponding target value. 

3. A method of inserting a watermark signal into video 
images as set forth in claim 2, where said positive value is 
+1 and said negative value is -1. 

4. A method of inserting a watermark signal into video 
images as set forth in claim 2, where said calculating step 
generates a zero value as the correction signal if the corre- 
sponding quantized DCT values of residual images are zero. 

5. A method of inserting a watermark signal into video 
images as set forth in claim 4, where said positive value is 
+1 and said negative value is -1. 

6. A method of inserting a watermark signal into video 
images comprising the steps of: 

receiving input video images; 

performing discrete cosine transformation (DCT) of said 
input video images to obtain DCT values of said input 
video images; 

adding watermark signals to said DCT values to obtain 

DCT values with watermark; 
performing discrete cosine transformation on predicted 

images to obtain DCT values of predicted images; 
subtracting DCT values of the predicted images from the 

DCT values with watermark to obtain DCT values of 

residual images; 
quantizing the difference values to obtain quantized DCT 

values of residual images; 
inverse quantizing the quantized DCT values at residual 

images and summing the inverse quantized DCT values 

of residual images with the DCT values of predicted 

images, for generating DCT coefficients for decoded 

images, 
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calculating correction signals from the DCT coefficients 
for decoded images; 

summing the correction signals with the quantized DCT 
values of residual images to obtain an output signal; 

inverse quantizing the output signal and inverse DCT the 
inverse quantized output signal and summing the 
resultant signal with predicted images to provide 
locally-decoded images; 

storing the locally-decoded images in memory for gener- 
ating the predicted images; and 

valuable length encoding the output signal to provide a 
watermarked MPEG video signal of the input video 
images. 

7. A method of inserting a watermark signal into video 
images as set forth in claim 6, where said calculating step 
generates a negative value as said correction signal when the 
extracted value exceeds the corresponding target value, and 
generates a positive value as said correction signal when the 
extracted value falls short of the corresponding target value. 

8. A method of inserting a watermark signal into video 
images set forth in claim 7, where said positive value is +1 
and said negative value is -1. 

9. A method of inserting a watermark signal into video 
images as set forth in claim 7, where said calculating step 
generates a zero value as the correction signal if the corre- 
sponding quantized DCT values of residual images are zero. 

10. A method of inserting a watermark signal into video 
images as set forth in claim 9, where said positive value is 
+1 and said negative value is -1. 



09/08/2004, EAST Version: 1.4.1 



United States Patent [19] 

Nakagawa et al. 



US006104826A 
[ii] Patent Number: 
[45] Date of Patent: 



6,104,826 
Aug. 15, 2000 



[54] METHOD OF WATERMARK-EMBEDDING/ 
EXTRACTING IDENTIFICATION 
INFORMATION INTO/FROM PICTURE 
DATA AND APPARATUS THEREOF, AND 
COMPUTER READABLE MEDIUM 

[75] Inventors: Akira Nakagawa; Kimihiko Kazui; 

Atsuko Tada; Elshi Morimatsu; Koich 
Tanaka, all of Kawasaki, Japan 

[73] Assignee: Fujitsu Limited, Kawasaki, Japan 

[21] Appl. No.: 08/948,083 

[22] Filed: Oct. 9, 1997 

[30] Foreign Application Priority Data 

Feb. 19, 1997 [JP] Japan 9-035258 

[51] Int. CI. 7 G06K 9/00 

[52] U.S. CI 382/100; 382/248 

[58] Field of Search 382/232, 100, 

382/238; 380/4, 34, 55, 23, 54, 287, 210 

[56] References Cited 

U.S. PATENT DOCUMENTS 

5,848,155 12/1998 Cox 382/191 

5,930,369 7/1999 Cox et al 380/54 

OTHER PUBLICATIONS 

EM. Boland J. J.K. O Rugnaidh and C. Dautzenberg "Water- 
marking Digital Images for Copyright Protection" Image 
Processing and its Applications Jul. 4-6 1995, Conference 
Publication No. 410 IEEE 1995 pp. 326-330, Apr. 1996. 
Takeshi Ogihara et al. "Data Embedding into Pictorial 
Images with less Distortion Using Discrete Cosine Trans- 
form" Proceedings of ICPR '96 1015-4651/96 IEEE 1996 
pp. 675-679. 

Yasuhiro Nakamura et al. A Unified Cooling Method of 
Image and Text Data Using Discrete Cosine Transition 
Systems and Computers in Japan, vol. 21, No. 3, 1990 pp. 
87-92, 1990. 

Adriang Bors et al "Image Watermarking Using DCT 
Domain Constraint" IEEE 1996 Department of Informatics, 
University of Thessaloniki Greece, pp. 231-234, 1996. 



I.J. Cox et al., "Secure Spread Spectrum Watermarking for 
Images, Audio and Video", 1996, pp. 243-246, Proceedings 
of the Int'l Conference of Image Processing, IEEE. 

JJ.K.O. Ruanaidh et al., "Watermarking Digital Images for 
Copyright Protection", Aug. 1996, vol. 143, No. 4, pp. 
250-256, IEE Proceedings: Vision, Image & Signal Pro- 
cessing, GB, Institute of Electrical Engineers, 

Kineo Matsui et al., "Video-Stenography: How to Secretly 
Embed a Signature in a Picture", Jan. 1994, vol. 1, No. 1, 
Jan. 1994, pp. 187-206, IMA Intellectual Property Project 
Proceedins. 

Jian Zhao et al., "Embedding Robust Labels Into Images for 
Copyright Protection", 1995, pp. 242-251, Proceedings of 
the Knowright. Conference. Proceedings of the Int'l Con- 
gress on Intellectual Property Rights for Specialized Infor- 
mation, Knowledge and New Technology. 

Primary Examiner — Andrew W Johns 
Assistant Examiner — Shervin Nakhjavan 
Attorney, Agent, or Firm—Siias & Halsey, LLP 

[57] ABSTRACT 

The present invention relates to methods for watermark- 
embedding/extracting identification information into/from 
picture data and apparatuses thereof. When identification 
information is watermark-embedded, the following steps are 
performed. Combinations of basis functions orthogonal each 
other are generated in association with each numerical 
signal. For each combination of the orthogonal basis 
functions, a weight coefficient is calculated so as to corre- 
spond to the combination of the basis functions by calcu- 
lating a sum of products of values of the basis functions for 
pixels in the original picture data and pixel values of the 
pixels. For each numerical signal, a watermark-embedding 
function is referred, a input value is specified, and a pixel 
value is changed. When the identification information is 
extracted from the picture data, for each weight coefficient, 
the watermark -embedding function is referred, and the value 
of the watermark-embedding function is calculated for each 
weight coefficient. 
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METHOD OF WATERMARK-EMBEDDING/ 
EXTRACTING IDENTIFICATION 
INFORMATION INTO/FROM PICTURE 
DATA AND APPARATUS THEREOF, AND 
COMPUTER READABLE MEDIUM 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to technology for 
watermark-embedding identification information indicating 
a person who has a right of copyright or the like into digital 
picture data distributed via various media such as circuits, 
and relates to technology for extracting identification infor- 
mation from those digital picture data. 

2, Description of the Related Art 

As digital technology and multi-media public advance in 
recent years, various data are converted into digital data and 
are distributed broadly via media such as a communication 
network, a satellite communication and a CD-ROM 
(Compact Disk Read Only Memory). When digital data are 
distributed in this multi-media public, it is not possible to 
avoid problems concerning copies of digital data. 

When a copy of digital data is lawful, it may help the 
culture in the multi-media public. When, however, the copy 
is unlawful such as used directly for business, there is a 
possibility that the copy causes an enormous profit loss for 
a person who has a right (such as an author, a copyrighter, 
a copyright holder, a person who has neighboring right) 
since it is possible to copy digital data without deterioration. 

Conventionally, technology is proposed in that identifi- 
cation information is watermark-embedded into picture data 
in a manner that it is difficult to recognize it by appearances, 
and the identification information is used as evidence when 
this picture data is copied unlawfully. For example, the 
following technology is proposed by J. Cox et al. in "Secure 
Spread Spectrum Watermarking for Multimedia", NEC 
Research Institute, Technical Report 95-10. That is, accord- 
ing to this technology, orthogonal transformation is applied 
to picture data, plural weight coefficients corresponding to 
dot positions of the identification information are selected 
among the weight coefficients of basis function obtained by 
the transformation, dot values of the identification informa- 
tion are respectively added to the selected weight 
coefficients, and the inverse orthogonal transformation is 
applied to all weight coefficients including the added 
coefficients, as the result, identification information 
watermark-embedded picture data is generated. 

In the above-mentioned conventional technology, 
however, since the weight coefficients obtained by the 
orthogonal transformation of original picture data (before 
watermark-embedding identification -information) may be 
various values, it is not possible to specify weight coeffi- 
cients to which dot values of the identification information 
are respectively added among the weight coefficients 
obtained by the orthogonal transformation of the identifica- 
tion information watermark-embedded picture data. Thus, to 
enforce the above-mentioned technology, it is necessary to 
keep and to manage each pair of original picture data and 
identification information watermark-embedded picture 
data, and when a copy appears, it is necessary to extract each 
dot value of identification information by subtracting each 
weight coefficient obtained by the orthogonal transformation 
of the kept and managed original data from each weight 
coefficient obtained by the orthogonal transformation of the 
kept and managed identification information watermark- 
embedded picture data and to extract each dot value of 
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identification information by subtracting each weight coef- 
ficient obtained by the orthogonal transformation of the 
original picture data from each weight coefficient obtained 
by the orthogonal transformation of the copy, thereafter, it is 
5 necessary to prove the identify about both identification 
information. 

According to the conventional technology, it is necessary 
to keep and to manage both identification information 
watermark-embedded picture data and original picture data 

10 twice, therefore, there are problems in that a data manage- 
ment and a proof to detect a copy are troublesome and twice 
storage are necessary compared with distributed data quan- 
tity. These problems are serious, particularly in a database in 
which data must be updated frequently, such as a database 

15 dealing picture data about 1000 pieces and a database for 
newspapers. 

To avoid this twice data management, it is also considered 
that a part of the weight coefficient obtained by the orthogo- 
nal transformation of the original data is permutated by the 

20 just value of the identification information. There are pos- 
sibilities in that this permutation causes a remarkable dete- 
rioration of the picture quality after the inverse orthogonal 
transformation, and in that the identification information is 
immediately recognized by a reproducer only by the 

25 orthogonal transformation of the picture data and then data 
is rewritten. 

SUMMARY OF THE INVENTION 

3 q Accordingly, it is an object of the present invention to 
provide a method and an apparatus for watermark- 
embedding identification information into picture data, 
wherein identification information can be extracted from 
identification information watermark -embedded picture data 

35 without original picture data and identification information 
can be watermark-embedded into picture data without dete- 
riorating the picture quality so as not to be recognized by a 
reproducer, to provide a method and an apparatus for 
extracting identification information from the picture data in 

40 which the identification information is watermark- 
embedded by those method and apparatus, and to provide a 
computer readable medium storing a program to make a 
computer function as an identification information 
watermark-embedding and/or extracting apparatus. 

45 The preset invention is achieved to solve the above- 
mentioned problems. 

That is, the first aspect of the present invention relates to 
an identification information watermark-embedding method 
for watermark-embedding identification information into 

50 original picture data consisting of pixel values which are 
arranged in a matrix. The numerical signals is not more than 
the pixel values. In the identification information 
watermark-embedding method, combinations of basis func- 
tions orthogonal each other are generated in association with 

55 each of the numerical signals (SI), weight coefficients are 
calculated so as that each of weight coefficients corresponds 
to each of the combinations of the basis functions by 
calculating, for each of the combinations of the basis 
functions, a sum of products, each of which are calculated 

60 for each of the pixels, based on a value of each of the basis 
functions for a position of the pixel within the original 
picture data and a pixel value of the pixel (S2), for each of 
the numerical signals, a watermark-embedding function 
which is a multi-to-one function taking available values of 

65 the weight coefficients in a domain and taking available 
values of the numerical signals in a range is referred, and a 
input value of the watermark-embedding function closest to 
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the weight coefficients corresponding calculated for the is obtained, it becomes easy to specify input values because 

combinations of the basis functions associated with the there are solutions at points that the intersection is shifted by 

numerical signal is specified among plural input values of times of whole numbers. Additionally, from points of the 

the watermark-embedding function outputting a numerical picture quality in the picture data after changing the pixel 

value of the numerical signal (S3), pixel values in the 5 values, it becomes subjectively difficult to watch errors by 

original data are changed so that each of the weight coef- this chan S e for components of which weight coefficients are 

ficients becomes a value equal to the input value which is lar S er - ^ chan S e d ^ ees are lar S er > resistance for various 

specified picture data change becomes stronger. To make the change 

P A . . , . #u . , . . degree large, a period of the watermark-embedding function 

According to the present invention, the numerical infer- & , , , 4 , , n 

; . L -j *• V - * a ' *i m may be made long. Thus, a watermark-embedding function 

mation of the identification information is not directly 10 7 , , . * . * k™™ c a ™~ r 

1 t j j j * * ii_ * i_* * * u . *u may be used in which the period becomes long in accor- 

watermark-embedded into the weight coefficient, but the , J , . lL t , /[, ... ™- • t u 

r . . . . »u * ■ i • u * « ■ * * dance with that the value of the weight coefficient becomes 
input value closest to the original weight coefficient is we . $ c (o a 

changed for the weight coefficient among the plural input { J ^ for a ^ coeffici and , ht f wei ht 

values which the numerical information is the output value B _ . , . . f, „„ , f „ „ „_ n„ „:„i.» 

. . , , ... - ,. _ .. i.-. ,c coefficient is changed to a small value for a smaller weight 

in the watermark-embedding function. Thus, the difference 3S coe gj c j en , 

of the weight coefficient before and after changing becomes coe cien. ... 

very small As the result, there is little case that the picture , A The second aspect of the present invention .relates to an 

quality of the picture data after changing the pixel value identification information extracting method for extracting 

deteriorates. Further, since the weight coefficient after identification ^formation from process object picture data 

changing the pixel value corresponds to one numerical value 20 into which identific aUon information is watermark- 

of the numerical information to be identification information tv ^f by the identification information watermark- 

by the watermark-embedding function, the identification embeAtaig method according to the first aspect. In the 

information can be extracted from the picture data after identification information extracting method, combinations 

changing the picture value. Moreover, though third persons of basis orthogonal each other is generated in 

calculate the weight coefficient, they can not detect which 25 association with each of the numerical signals For each of 

value corresponds to each weight coefficient without the the combinations of the orthogonal bans functions, weight 

i u aa- a r tu,,, ,;„™ ( k fl ti,;^ coefficients are calculated so that each of weight coefficients 

watermark-embedding function. Thus, since the third per- f . 

sonscannotknowcontentsoftheidentificationinfonnation, corresponds to each of the combinations of the basis func- 
they can not modify the identification information. calculating a sum of products each of which are 

JL . , . . , . . , . , , ™ calculated, for each of the pixels, based on a value or each 

The pixel value in the original picture data may be a Qf ^ ^ for a ^ Qf ^ ^ ^ ^ 

uminance value of each color signal in RGB signal, a ori ^ na , kme ^ ^ a ixe , yalue of , he kel ^ for 

luminance yalue in YCC signal or a color difference value in ej * Qf ^ weigh( coefficients> a wa , ermark . ern bedding 

YCC signal. function which is a multi-to-one function taking available 

The dot number in the original picture data may be set so 35 values of me weight coe fficients in a domain and taking 

as to be equal or different lengthwise and breadthwise. available values of the numerical signals in a range is 

The available numerical value of each numerical value referred and a value of the watermark-embedding function 

signal in the identification information may be binary or f or eac h 0 f the weight coefficients is calculated, 
more than. Th e m j rc i aspect of the present invention relates to an 

The numerical signals in the identification information 40 identification information watermark-embedding method 

may be arranged in a line or may be picture information for watermark-embedding identification information into 

arranged in a matrix. original picture data consisting of pixel values which are 

The weight coefficient may be calculated only for a arranged in a matrix. The identification information consists 

combination of basis functions corresponding to each of numerical signals not more than the pixel values. In the 

numerical signal in the identification information, or may be 45 identification information watermark-embedding method, 

calculated for all combinations of basis functions by the orthogonal transformation is applied to each of pixel values, 

orthogonal transformation. According to the former, the total and coefficient distributed data consisting of weight coeffi- 

of the calculation process for the weight coefficient reduces. cients which are arranged in a matrix is generated, each of 

According to the latter, it is possible to change the pixel weight coefficients selected from the coefficient distributed 

value of the picture data only by applying the inverse 50 data is related with one of the numerical signals. For each of 

orthogonal transformation to all weight coefficient after the numerical signals, a watermark-embedding function 

permutating the weight coefficient for the combination of the which is a multi -to-one function taking available values of 

basis functions corresponding to each numerical signal in the weight coefficients in a domain and taking available 

the identification information. As this orthogonal values of the numerical signals in a range is referred, an 

transformation, a two-dimensional discrete cosine 55 input value of the watermark-embedding function to the 

transformation, a two-dimensional discrete sine weight coefficients related with the numerical signal is 

transformation, or a Hadamard transformation may be used. specified among plural input values of which output values 

The watermark-embedding function may be held by func- correspond to a numerical value of one of the numerical 

tional expressions or by tables providing relations of one signal, and the weight coefficients in the coefficient distrib- 

output value for plural input values. The watermark- 60 uted data related with the numerical signal is permutated by 

embedding function may be a periodic function or not, as the input value. Then, inverse orthogonal transformation is 

long as a multi-one function is used. When a periodic applied to the coefficient distributed data after permutating 

function is used, functional expressions become simple, the weight coefficients for all of the numerical signals, 
therefore, it is possible to simplify an apparatus to carry out The fourth aspect of the present invention relates to an 

the present invention. That is, only if a intersection with a 65 identification information extracting method for extracting 

value of each numerical signal of the identification infor- identification information from process object picture data to 

mation in one period of the watermark-embedding function which the identification information is watermark- 
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embedded by the identification information watermark- 
embedding method according to the third aspect. In the 
identification information extracting method, orthogonal 
transformation is applied to each pixel values and coefficient 
distributed data consisting of weight coefficients which are 5 
arranged is generated in a matrix, weight coefficients cor- 
responding to each of the numerical signals are picked up 
from the coefficient distributed data. For each of the weight 
coefficients which are picked up, a watermark-embedding 
function which is a multi-to-one function taking available 1Q 
values of the pickup weight coefficients in a domain and 
taking available values of the numerical signals in a range is 
referred, and a value of the watermark-embedding function 
for the each of the weight coefficients which are picked up 
is calculated. 35 

The fifth aspect of the present invention relates to an 
identification information watermark-embedding method 
for watermark-embedding identification information into 
original picture data consisting of pixel values which are 
arranged in a matrix. The identification information consists 2 q 
of numerical signals not more than the pixel values. In the 
identification information watermark-embedding method, 
two-dimensional discrete cosine transformation is applied to 
each of pixel values, and generating coefficient distributed 
data consisting of weight coefficients which are arranged in 25 
a matrix, each of weight coefficients selected from the 
coefficient distributed data is related with one of the numeri- 
cal signals. For each of the numerical signals, a watermark- 
embedding function which is a multi- to-one function taking 
available values of the selected weight coefficients in a 30 
domain and taking available values of the numerical signals 
in a range is referred, an input value closest to the weight 
coefficients related with the numerical signals is specified 
among plural input values which output values correspond 
to a numerical value of the numerical signal, and the weight 35 
coefficients in the coefficient distributed data related with the 
numerical signal is permutated by the closest input value. 
Then, inverse two-dimensional discrete cosine transforma- 
tion is applied to the coefficient distributed data after per- 
mutating the weight coefficients for all of the numerical ^ 
signals. 

The sixth aspect of the present invention relates to an 
identification information extracting method for extracting 
identification information from process object picture data 
into which the identification information is watermark- 45 
embedded by the identification information watermark- 
embedding method according to the fifth aspect. In the 
identification information extracting method, two- 
dimensional discrete cosine transformation is applied to 
each pixel values, and co efficient distributed data consisting 50 
of weight coefficients which are arranged in a matrix is 
generated. Weight coefficients corresponding to the numeri- 
cal signals are picked up from the coefficient distributed 
data. For each of the weight coefficients which are picked 
up, a watermark-embedding function which is a multi-to- 55 
one function taking available values of the weight coeffi- 
cients in a domain and taking available values of the 
numerical signals in a range is referred, and a value of the 
watermark-embedding function for the each of the weight 
coefficients which are picked up is calculated. 6 q 

The seventh aspect of the present invention relates to an 
identification information watermark-embedding method 
according to the first, third, or fifth aspect, and is specified 
by that the watermark-embedding function is a period func- 
tion. 65 

The eighth aspect of the present invention relates to an 
identification information extracting method according to 



the second, fourth, or sixth aspect, and is specified by that 
the watermark-embedding function is a period function. 

The ninth aspect of the present invention relates to an 
identification information watermark-embedding method 
according to the first, third, or fifth aspect, and is specified 
by that the watermark-embedding function is a continuous 
period function. 

The tenth aspect of the present invention relates to an 
identification information extracting method according to 
the second, fourth, or sixth aspect, and is specified by that 
the watermark-embedding function is a continuous period 
function. 

The eleventh aspect of the present invention relates to an 
identification information watermark-embedding method 
according to the first, third, or fifth aspect, and is specified 
by that the watermark-embedding function makes an inter- 
val between the plural input values taking one output value 
narrow when an input value is low, and makes the interval 
broad when the input value is high. 

The twelfth aspect of the present invention relates to an 
identification information extracting method according to 
the second, fourth, or sixth aspect, and is specified by that 
the watermark-embedding function makes an interval 
between the plural input values taking one output value 
narrow when an input value is low, and makes the interval 
broad when the input value is high. 

The thirteenth aspect of the present invention relates to an 
identification information watermark-embedding apparatus 
of watermark-embedding identification information into 
original picture data consisting of pixel values which are 
arranged in a matrix. The identification information consists 
of numerical signals not more than the pixel vales. The 
identification information watermark-embedding apparatus 
comprises a watermark-embedding function hold unit for 
holding a watermark -embedding function which is a multi - 
to-one function taking available values of the weight coef- 
ficients in a domain and taking available values of the 
numerical signals in a range, an orthogonal transformation 
unit for applying orthogonal transformation to each of pixel 
values, and generating coefficient distributed data consisting 
of weight coefficients which are arranged in a matrix, a 
weight coefficient permutation unit for relating each of 
weight coefficients selected from the coefficient distributed 
data with one of the numerical signals, specifying an input 
value closest to the weight coefficients related with the 
numerical signals among plural input values of the 
watermark-embedding function outputting a numerical 
value of the numerical signal, and permutaling the weight 
coefficient in the coefficient distributed data by the closest 
input value, and an inverse orthogonal transformation unit 
for applying inverse orthogonal transformation to the coef- 
ficient distributed data which the weight coefficients are 
permutated. 

The fourteenth aspect of the present invention relates to 
an identification information extracting apparatus for 
extracting identification information from process object 
picture data into which the identification information is 
watermark-embedded by the identification information 
watermark-embedding apparatus according to the thirteenth 
aspect. The identification information extracting apparatus 
comprises a watermark-embedding function hold unit for 
holding a watermark-embedding function which is a multi- 
to-one function taking available values of the weight coef- 
ficients in a domain and taking available values of the 
numerical signals in a range, an orthogonal transformation 
unit for applying orthogonal transformation to each of pixel 
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values and generating coefficient distributed data consisting 
of weight coefficients which are arranged in a matrix, a 
pickup unit for picking up weight coefficients corresponding 
to each of the numerical signals from the coefficient distrib- 
uted data, and a calculation unit for calculating a value of the 5 
watermark-embedding function for each of the weight coef- 
ficients which are picked up. 

The fifteenth aspect of the present invention relates to a 
computer readable medium storing a program to control a 
computer. The program performs a step of applying orthogo- 10 
nal transform to each of pixel values and generating coef- 
ficient distributed data consisting of weight coefficients 
which are arranged in a matrix; a step of relating each of 
weight coefficients selected from the coefficient distributed 
data with the numerical signals; a step of, for each of the is 
numerical signals, referring to a watermark-embedding 
function which is a multi-to-one function taking available 
values of the weight coefficients in a domain and taking 
available values of the numerical signals in a range, speci- 
fying an input value closest to weight coefficients related 20 
with the numerical signals among plural input values of the 
watermark-embedding function outputting a numerical 
value of the numerical signal, and permutating the weight 
coefficients in the coefficient distributed data by the closest 
input value; and a step of applying inverse orthogonal 25 
transformation to the coefficient distributed data after per- 
mutating the weight coefficients for all of the numerical 
signals. 

The sixteenth aspect of the present invention relates to a 
computer readable medium storing a program to control a 30 
computer, The program performs a step of applying orthogo- 
nal transform to each pixel values, and generating coefficient 
distributed data consisting of weight coefficients which are 
arranged in a matrix; a step of picking up weight coefficients 
corresponding to the numerical signals from the coefficient 35 
distributed data; and a step of, for each of the weight 
coefficients which are picked up, referring to a watermark- 
embedding function which is a multi-to-one function taking 
available values of the pickup weight coefficients in a 
domain and taking available values of the numerical signals 40 
in a range and calculating a value of the watermark- 
embedding function for the each of the pickup weight 
coefficients. 

BRIEF DESCRIPTION OF THE DRAWINGS 45 

Other objects and advantages of the present invention will 
become apparent during the following discussion conjunc- 
tion with the accompanying drawings, in which: 

FIG. 1 is a view illustrating a principal of the present 
invention; 50 

FIG. 2 is a block diagram illustrating an outline structure 
of a watermark-embedding computer according to the aspect 
of the present invention; 

FIG. 3 is a flow chart illustrating an identification infor- 55 
mation watermark-embedding process executed by the CPU 
1 in FIG. 2; 

FIG. 4 is a block diagram illustrating an outline structure 
of an extracting computer according to the aspect of the 
present invention; 60 

FIG. 5 is a flow chart illustrating an identification infor- 
mation extracting process performed by the CPU 1 in FIG. 
4; 

FIG. 6 is a flow chart illustrating an identification infor- 
mation watermark-embedding process in Embodiment 1; 55 

FIG. 7 is a flow chart illustrating a two-dimensional DCT 
process subroutine executed in S204 in FIG. 6; 



FIG. 8 is a flow chart illustrating a two-dimensional DCT 
process subroutine executed in S213 in FIG. 6; 

FIG. 9 is a flow chart illustrating an identification infor- 
mation extracting process in the Embodiment; 

FIGS. 10(a) through 10(/i) are explanatory views illus- 
trating a flow of a signature picture watermark-embedding 
process in the Embodiment; and 

FIG. 11(a) through 11(e) are explanatory views illustrat- 
ing a flow of a signature picture extracting process in the 
Embodiment. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

Preferred aspects and embodiments of the present inven- 
tion will now be described with reference to the accompa- 
nying drawings. 

A watermark-embedding computer according to an aspect 
of a method and an apparatus for watermark-embedding 
identification information into image data and an extracting 
computer according to an aspect of a method and an appa- 
ratus for extracting identification information from image 
data according to the present invention are structured in a 
manner that conventional various orthogonal transformation 
processes and inverse orthogonal transformation processes 
can be used, and any original picture and identification 
information of various sizes can be processed (only, iden- 
tification information ^original picture). 
[Outline of Identification Information Watermark- 
embedding and Extracting] 

First, explanations are given of outlines of watermark- 
embedding identification information into picture data and 
of extracting identification information from into identifica- 
tion information watermark-embedded picture data before 
explaining a concrete structure in this aspect. 

Now, it is assumed that original picture data (such as 
monochrome picture data or luminance data extracted from 
NTSC color picture data) to be an identification information 
watermark-embedded object is structured by NxN pixels. It 
is also assumed that the identification information to be a 
watermark object is data including L (L=NxM) signals to 
which predetermined values are respectively given. 

The watermark-embedding computer applies the orthogo- 
nal transformation to all of the above-mentioned original 
picture data or to each of plural blocks obtained by dividing 
the original picture data (only, not less than L pixels in each 
block). The original picture is divided and the orthogonal 
transformation is applied to those, because a signature 
picture can be extracted without deterioration though a 
picture to which a watermark-embedding process is applied 
is cut off partially and copied. Incidentally, when a size of 
each partial picture is made too small, the picture to which 
the watermark-embedding process is applied deteriorates, 
therefore, it is preferable that each partial picture is not less 
than 8x8 pixels. 

For the orthogonal transformation, L combinations of 
basis functions which are orthogonal one another in relation 
to each signal of the identification information are prepared. 
Each weight coefficient of combinations of basis functions is 
obtained by calculating a sum of products of a value of each 
basis function orthogonal to a position of each pixel in the 
original picture data and a luminance value thereof for each 
combination of basis functions. Incidentally, "orthogonal" 
indicates that directions of variables of each basis function 
are orthogonal one another in an original picture or a block 
including NxM pixels. 

Then, the watermark-embedding computer watermarks 
each value of the signal included in the identification infor- 
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mation into each weight coefficient for the combination of 
basis functions which are previously related. At that time, 
the watermark-embedding computer does not directly add/ 
write a value of each signal to/over a weight coefficient, but 
prepares a predetermined watermark-embedding function 
every signal and permutates a value corresponding to each 
signal for an original weight coefficient by this watermark- 
embedding function. 

The watermark-embedding function prepared every sig- 
nal may be defined so as to be different every signal or may 
be defined in common for signals. In any case, each 
watermark-embedding function includes a value of a weight 
coefficient within a defined range, and includes a value of 
each signal in the identification information within a numeri- 
cal range. Each watermark-embedding function is a multi- 
one function in which plural input values in the defined 
range correspond to the same output value. Concretely, a 
process becomes simple when a simple function is used, 
therefore, it is desirable to use a periodic function (a 
continuous periodic function in which an interval between 
plural input values to be the same output value is narrow 
when the input value is small and the interval is broad when 
the input value is large) indicated by the following expres- 
sion (1) and so on. 



(1) 



The watermark-embedding computer, every signal in the 
identification information, inverse-calculates all input val- 
ues in the watermark-embedding function in which a value 
of that signal is taken for a output value. Then, the 
watermark-embedding computer specifies an input value 
which has the smallest difference from the weight coefficient 
of the combination of the basis functions previously corre- 
sponded to the signal among all calculated input values, and 
permutates the specified input value for the original weight 
coefficient value. 

Thereafter, the watermark-era bedding computer rewrites 
a value of each pixel in the original picture data or in the 
block in a manner that each original weight coefficient of the 
combination of basis functions after the orthogonal trans- 
formation becomes a permutated value. Identification infor- 
mation watermark-embedded picture data is obtained in this 
way. As above mentioned, since the variation by permutat- 
ing the weight coefficient is limited to the minimum, the 
deterioration of the identification information watermark- 
embedded picture data is kept to the minimum. Further, 
although a third person applies the orthogonal transforma- 
tion to the identification information watermark-embedded 
picture data, the person can not know each signal value of 
the identification information in a case that the watermark- 
embedding function is not known, therefore, it is not pos- 
sible to change the identification information. 

The extracting computer applies the orthogonal transfor- 
mation to all of process object picture data or to each of 
plural blocks obtained by dividing the process object picture 
data (only, not less than L pixels in each block). At that time, 
as well as for the case that the identification information is 
watermark-embedded into the original picture data, L com- 
binations of basis functions which are orthogonal to one 
another in relation to each signal of the identification infor- 
mation are available. Each weight coefficient of the combi- 
nation of basis functions is obtained by calculating a sum of 
products of respective values of basis functions orthogonal 
to positions of pixels in the process object picture data and 
luminance values thereof for combinations of basis func- 
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tions. Incidentally, as basis functions, the same functions for 
watermark-embedding the identification information into 
the original picture data are used. 

The extracting computer inputs each obtained weight 
coefficient into the corresponding watermark-embedding 
function so as to obtain an output value. Then, each obtained 
output value of the watermark-embedding function is 
arranged in accordance with the order of the signals in the 
identification information which are previously related with 
the corresponding weight coefficients. Then, when the pro- 
cess object picture data is the identification information 
watermark-embedded picture data, the arranged output val- 
ues coincide with the identification information. In this way, 
the extracting computer can extract the identification infor- 
mation from the identification information watermark- 
embedded picture data without original picture data only 
with each watermark-embedding function. The watermark- 
embedding function may be used for various original picture 
data in common. Thus, the total of data to be kept and 
managed reduces exceedingly compared with the conven- 
tional approach. 

[Structure of Watermark-embedding Computer] 

Next, an explanation is given of a concrete structure of the 
watermark -embedding computer. FIG. 2 is an outline block 
diagram illustrating only a structure relative to a process for 
watermark-embedding identification information into origi- 
nal picture data in hardware of this watermark-embedding 
computer. As shown in FIG. 2, the watermark-embedding 
computer is provided with a CPU (Central Processing Unit) 
1, an input disk unit 2, a ROM (Read Only Memory) 3 and 
an output disk unit 4 which are connected one another by a 
bus B. 

The input disk unit 2 inputs original picture data 21 into 
the CPU 1 in accordance with an instruction from the CPU 
1, such as a hard disk unit, a floppy disk unit, or an optical 
magnetic disk unit. 

The ROM 3 used as a watermark-embedding function 
hold unit and a computer readable medium is a read only 
memory holding an identification information watermark- 
embedding program 31 executed in the CPU 1, identification 
information 32 and a watermark-embedding function 33, 

The CPU 1 is a processor controlling all of the watermark- 
embedding computer. The identification information 
watermark-embedding program 31 read from the ROM 3 
runs, whereby an orthogonal transformation process part 11, 
a coefficient extraction process part 12, a coefficient permu- 
tation part 13, a coefficient watermark-embedding part 14 
and an inverse orthogonal transformation part 15 are imple- 
mented in the CPU 1, and the identification information 
watermark-embedding process shown in FIG. 3 is executed, 
Dot lines in FIG. 2 indicate data flows in the CPU 1. 

The orthogonal transformation process part 11 used as an 
orthogonal transformation unit applies the orthogonal trans- 
formation to the original picture data 21 of NxM pixels read 
from the input disk unit 2, and then calculates a weight 
coefficient for each of NxM combinations of the basis 
functions. The NxM weight coefficients forms a NxM 
matrix similarly to the original picture. The matrix of those 
weight coefficients, hereinafter, is called "coefficient distrib- 
uted picture data" for convenience. The orthogonal trans- 
formation process part 11 informs both the coefficient 
extraction process part 12 and the coefficient watermark- 
embedding part 14 of this coefficient distributed picture data. 

The coefficient extraction process part 12 extracts L 
weight coefficients to which signals of identification infor- 
mation are respectively watermark-embedded from the coef- 
ficient distributed picture data informed from the orthogonal 
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transformation process part 11, and informs the coefficient 
permutation part 13 of them. 

The coefficient permutation part 13 used as a coefficient 
permutation unit reads the identification information 32 and 
the watermark-embedding function 33 from the ROM 3. 
Then, the watermark-embedding function 33 is inverse - 
calculated every signal of the identification information 32, 
and then all input values, which take the output value for the 
value of the signal, of the watermark-embedding function 33 
are obtained. Then, the coefficient permutation part 13 
specifies an input value which has the smallest difference 
from the corresponding weight coefficient informed from the 
coefficient extraction process part 12 among the obtained 
input values, and informs the coefficient watermark- 
embedding part 13 of the specified input value as a rewrite 
value for that coefficient. 

The coefficient watermark-embedding part 14 used as a 
coefficient watermark-embedding unit writes this rewrite 
value over the weight coefficient corresponding to the 
rewrite value informed from the coefficient watermark- 
embedding part 13 in the coefficient distributed picture data 
received from the orthogonal transformation process part 11. 

The inverse orthogonal transformation part 15 used as an 
inverse orthogonal transformation unit applies the inverse 
orthogonal transformation to the NxM weight coefficients 
received from the coefficient watermark-embedding part 14, 
The inverse orthogonal transformation part 15, every pixel 
in the identification information watermark-embedded 
picture, calculates a total of products of each weight coef- 
ficient corresponding to each combination of the basis 
functions used in the orthogonal transformation part 11 and 
each value of basis function corresponding to the position of 
that pixel, and then obtains a luminance value of that pixel. 
The inverse orthogonal transformation part 15 sends an 
identification information watermark-embedded picture data 
41 including each pixel of which the luminance value is 
obtained like this to the output disk unit 4. 

In the output disk unit 4, such as a hard disk unit, a floppy 
disk unit and an optical magnetic disk unit, the identification 
information watermark -embedded picture data 41 received 
from the CPU 1 is written. 

FIG. 3 is a flow chart illustrating an identification infor- 
mation watermark-embedding process performed by the 
CPU 1 which has read the identification information 
watermark-embedding program 31. The identification infor- 
mation watermark-embedding process starts by inputting an 
identification information watermark-embedding command 
via a keyboard (not shown) connected to the watermark- 
embedding computer. 

In S001 executed initially after starting the identification 
information watermark-embedding process, the CPU 1 reads 
L signals S ( - [i=l-L] to be in the identification information 32 
from the ROM 3. 

In S002, the CPU 1 reads L watermark -embedding func- 
tions 33 (f ( <x)[i-l-L]) from the ROM 3. 

In S003, the CPU 1 reads original picture data of NxM 
pixels from the input disk unit 2. 

In S004, the CPU 1 applies the above mentioned orthogo- 
nal transformation process to all of the original picture data 
read in S003 so as to generate coefficient distributed picture 
data of NxM dots. 

In S005, the CPU 1 selects a weight coefficient Q[i=l-L] 
corresponding to each signal S t {i=l-L] read in S001 among 
weight coefficients included in the coefficient distributed 
picture data generated in S004. 

In S006, the CPU 1 initializes a variable i to specify the 
weight coefficient Q to be a process object to "1". 
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Next, the CPU 1 executes a loop process between S007 
and S010. In S007 to be the first step of this loop, the CPU 
1 obtains K-piece solutions c^ (l<k<K,) which satisfy the 
relation f^O^S,. based on the identification information 
5 signal S f corresponding to the weight coefficient C of the 
process object and the watermark-embedding function f,(x). 
In this case, it is defined that a number of solutions c rt for 
the watermark-embedding function f/x^S,- prepared corre- 
spondingly to the ith signal S,- is K,-. 

In S008, the CPU 1 rewrites a weight coefficient Q of the 
process object in the coefficient distributed picture data 
generated in S004 by a solution closest to the weight 
coefficient Q among solutions c ik obtained in S007. 

In S009, the CPU 1 checks whether the variable i gets to 
L or not, namely, whether processes in S007 and S008 are 
35 applied to all signals included in the identification informa- 
tion. Then, when the -variable i does not get to L, the CPU 
1 increases the variable i, and then returns the operation to 
the S007. 

On the contrary, when the variable i gets to L, the CPU 1 

20 applies the above-mentioned inverse orthogonal transforma- 
tion to the coefficient distributed picture rewritten in S008 so 
as to generate identification information watermark- 
embedded picture data of MxM dots. 

In S012, the CPU 1 writes the identification information 

25 watermark-embedded picture data generated in SOU into the 
output disk unit 4. Then, the CPU 1 terminates the 
watermark-embedding process. 
[Structure of Extracting Computer] 

Next, an explanation is given of a concrete structure of the 

30 extracting computer. FIG. 4 is an outline block diagram 
illustrating only a structure relative to a process for extract- 
ing identification information from the identification infor- 
mation watermark-embedded picture data in hardware of 
this extracting computer. As shown in FIG. 4, the extracting 

35 computer is provided with a CPU (Central Processing Unit) 
1, an input disk unit 2, a ROM (Read Only Memory) 3 and 
an output unit 5 which are connected one another by a bus 
B. That is, the hardware configuration of the extracting 
computer is similar to that of the watermark-embedding 

40 computer, however, there are some differences that an iden- 
tification information extracting program 34 is stored in the 
ROM 3 and the output unit 5 is necessary but the output disk 
unit 4. Thus, when both the identification information 
watermark-embedding program 31 and the identification 

45 information extracting program 34 are stored in the ROM 3, 
and the output disk unit 4 and the output unit 5 are connected 
to the bus B. one computer can be used as the watermark- 
embedding computer and the extracting computer. 

In FIG. 4, the input disk unit 2 inputs process object 

50 picture data (identification information watermark- 
embedded picture data) 22 into the CPU 1, such as a hard 
disk unit, a floppy disk unit and an optical magnetic disk 
unit. 

The ROM 3 used as a watermark function hold unit and 
55 a computer readable medium is a read only memory holding 
an identification information extracting program 34 running 
in the CPU 1 and the identification information 32. This 
identification information 32 is just the same in the 
watermark-embedding computer. 
60 The CPU 1 is a processor controlling all of the extracting 
computer. The identification information extracting program 
34 read from the ROM 3 runs, whereby an orthogonal 
transformation process part 16, a coefficient extraction pro- 
cess part 17 and an identification information calculation 
65 part 18 are implemented in the CPU 1, and the identification 
information extracting process shown in FIG, 5 is executed. 
Dot lines in FIG. 4 indicate data flows in the CPU 1. 
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The orthogonal transformation process part 16 used as an 
orthogonal transformation unit applies the above-mentioned 
orthogonal transformation to the process object picture data 
22 of NxM pixels, read from the input disk unit 2, and then 
calculates a weight coefficient for each of NxM combina- 5 
tions of the basis functions. The orthogonal transformation 
process part 16 informs the coefficient extraction process 
part 17 of this coefficient distributed picture data including 
NxM weight coefficients. 

The coefficient extraction process part 17 as a pickup unit 10 
extracts L-piece weight coefficients in which there is a 
possibility that signals of the identification information are 
watermark-embedded, and then informs the identification 
information calculation part 18 of them. 

The identification information calculation part 18 used as is 
a calculation unit reads a watermark-embedding function 33 
from the ROM 3. The output value of the watermark- 
embedding function 33 is obtained every weight coefficient 
informed from the coefficient extraction process part 17. 
Then, the identification information calculation part 18 20 
arranges each obtained output value in accordance with the 
arrangement in the coefficient distributed picture data of the 
weight coefficient corresponding to that output value, and 
then outputs them to the output unit 5. 

The output unit 5 is a display unit for displaying L-piece 25 
output values received from the CPU 1, a printer for printing 
output values or the like. 

FIG. 5 is a flow chart illustrating an identification infor- 
mation extracting process executed by the CPU 1 which has 
read the identification information extracting program 34. 30 
The identification information extracting process starts by 
inputting an identification information extracting command 
via a keyboard (not shown) connected to the watermark- 
embedding computer. 

In S101 executed initially after starting the identification 35 
information extracting process, the CPU 1 reads L-piece 
watermark-embedding functions 33(f/x) [i=l-L]) from the 
ROM 3. 

In S102, the CPU 1 reads the process object picture data 
of NxM pixels from the input disk unit 2. 40 

In S103, the CPU 1 applies the above mentioned orthogo- 
nal transformation process to all of the process object picture 
data read in S102 so as to generate coefficient distributed 
picture data of NxM dots. 

In S104, the CPU 1 selects a weight coefficient C, [i=l-L] 45 
at a position corresponding to each signal S ( . [i=l-L] of the 
identification information 32 in the watermark-embedding 
computer among weight coefficients included in the coeffi- 
cient distributed picture data generated in S103. 

In S105, the CPU 1 initializes a variable i to specify the 50 
weight coefficient C,- of a process object into "1". 

Next, the CPU 1 executes a loop process between S106 
and S108. In S106 to be the first step of this loop, the CPU 
1 obtains an output value S f . for the weight coefficient Q of 
the watermark-embedding function f/x) corresponding to 55 
the weight coefficient C of the process object. 

In S107, the CPU 1 checks whether the variable i gets to 
L or not, namely, whether the process in S106 is applied to 
all weight coefficients to which identification information 
may be watermark-embedded or not. When the variable i 60 
does not yet get to L, the CPU 1 increases the variable i in 
S108, and then returns the process to the S106. 

On the contrary, when the variable i gets to L, in S109, the 
CPU 1 arranges all output values S,- [i=l-L] obtained in 
S106 in accordance with the arrangement in the coefficient 65 
distributed picture data of the corresponding weight coeffi- 
cients C ( - [i-l-L] so as to output them to the output unit 5. 
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As the results, the output unit 5 can display or print data 
corresponding to the identification information when the 
process object picture data read in S102 is identification 
information watermark-embedded picture data. 

Additionally, two-dimensional DCT (Discrete Cosine 
Transformation), two-dimensional DST (Discrete Sine 
Transformation) or two-dimensional Hadamard's Transfor- 
mation maybe used as the orthogonal transformation in this 
aspect. Next, an explanation will be given of the concrete 
processes in the identification information watermark- 
embedding process and the identification information 
extracting process when two-dimensional DCT is used as 
the orthogonal transformation with reference to Embodi- 
ment 1. 

Embodiment 

In this Embodiment, an original picture data includes 
NxN pixels (i.e., square of pikets; only, N»8), as shown in 
FIG. 10(a). The luminance value of each pixel in the original 
picture data is shown by the 0-255 gray scale. The identi- 
fication information, as shown in FIG. 10(d), is picture data 
(hereinafter, called "signature picture data") showing char- 
acters "FJ" by giving a white luminance value (255) or a 
black luminance value (0) selectively to each of 8x8 pixels. 
In addition to this, the identification information watermark- 
embedded picture data is called "signature picture 
watermark-embedded picture data". Since the weight coef- 
ficient obtained by the two-dimensional DCT corresponds to 
the intensity (amplitude) of each frequency component in 
the original picture data, the above-mentioned "coefficient 
distributed picture data" is called "frequency distributed 
picture data". As the watermark-embedding function, one 
function f(x) applied to respective weight coefficients in 
common, namely, a continuous periodic function in a saw 
tooth shown in a graph in FIG. 10(e) is used. 

(Identification Information Watermark-embedding Process) 
FIGS. 6 and 8 are flow charts illustrating the identification 
information watermark-embedding process in the Embodi- 
ment. 

In S201 executed initially after staring the identification 
information watermark-embedding process, the CPU 1 reads 
signature picture data S (i j) [i-O-7, j— 0— 7]) from the ROM 
3. 

In S202, the CPU 1 reads the watermark-embedding 
function f(x) from the ROM 3. 

In S203, the CPU 1 reads the process object picture data 
of NxN pixels from the input disk unit 2. 

In $204, the CPU 1 applies the two-dimensional DCT to 
all of the original picture data read in S203, and then 
generates frequency distributed picture data of NxN dots 
shown in FIG. 10(&). Concretely, in S204, the CPU 1 
performs a two-dimensional DCT process subroutine shown 
in FIG. 7. 

In S301 to be the first step of this two-dimensional DCT 
process subroutine, the CPU 1 initializes a variable i show- 
ing a column (0 is leftmost) in the frequency distributed 
picture data of the calculation object weight coefficient C(i j) 
to be "0". 

In S302, the CPU 1 sets a function W,(x) of a variable x 
as indicated by the following expression (2). In this 
expression, the variable x corresponds to a column (0 is 
leftmost) of each pixel in the original picture data. 
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and to watermark the signature picture data into those areas, 

ry (2) The signature picture data is watermark-embedded into only 

w 0 W = J — weight coefficients indicating DC components and weight 

N coefficients indicating low frequency components, because 

5 some variations of the DC components and low frequency 

In S303, the CPU 1 initialize a variable j showing a row components influence slightly on the picture quality of the 

(0 is highest) in the frequency distributed picture data of the signature picture watermark-embedded picture data, 
calculation object weight coefficient C(i j) to be "0". in S205, the CPU 1 initializes the variable i indicating a 

In S304, the CPU 1 sets a function Wfy) of a variable y column of the pickup object weight coefficient C(ij) in the 

as indicated by the following expression (3). In this i 0 frequency distributed picture data and indicating a column 

expression, the variable y corresponds to a row (0 is highest) of thc re f cr ence object pixel S(ij) in the signature picture so 

of each pixel in the original picture data). as t0 ^ e «q» 

In S206, the CPU 1 initializes the variable j indicating a 

w IT ^ row of the pickup object weight coefficient C(ij) in the 

~ V 75 35 frequency distributed picture data and indicating a row of 

the reference object pixel S(i j) in the signature picture so as 

In S305, the CPU 1 executes the following expression (4) t °I )C c ! ) ft , . „„ TT i , . . ^ . 

based on both functions W,{x) and Wfy) set at the current , A ^207 the CPU 1 reads the signature image data (FIG. 
time and a luminance value G(x,y) of each pixel in the 20 J ««* «J« watermark-embedding function f(x) (FIG. 

original picture data, and sets the calculated series as the 10 ( e )) from the R0M 3 > and then obtains aU in P ul values c o* 

weight coefficient C(i,j) specified by the current variables i of the watermark-embedding function f(x) which takes a 

anc j j. luminance value of the reference object pixel S(ij) in the 

signature picture data specified by the current variables i and 

N .i N .i (4) j for an output value. That is, all of K^-piece solutions c ijk 

;) = Yj £ c <*- • ^ w jW 25 (! = k = ^sy) which satisfy ffe^Sflj) are obtained. Here, it 

*-° is defined that the number of the solutions c ijlc for f(c tyfe )=S 

(i j) * K, y . 

In S306, the CPU 1 checks whether the current variable ^ S2 °? > the 5 PU 1 f^ts an J a P ut ™f™ which is closest 

j gets to (N-l) or not. When the variable j does not yet get 30 t0 the P icku P ob J ect wei ^ hl ^^soX C(i,j) specified by the 

to (N-l), thc CPU 1 increases the variable j in S307. Then, currenl variables 1 and j among all input values c„ A obtained 

the CPU 1 substitutes the current variable j to the following in s207 - the CPU 1 permutates the pickup object 

expression (5), and sets a new function W ( <y) of a variable wei S ht coefficient C(i,j) specified by the current variables 1 

y a g a i n# and j for the selected input value (see FIG. 10(f)). 

35 In S209, the CPU 1 checks whether the current variable 

j — (5) J g ets t0 or not - When the variable j does not yet get to 

Wj(y) =J- cm** 2 *" 1 " 0 ; "7", the CPU 1 increases the variable j in S210, and then 

' N 2N returns the process to S207 to permutate the weight coeffi- 
cient C(ij) in -the next row. 

Thereafter, the CPU 1 returns the process to S305, and 40 On the contrary, when it is determined that the current 

calculates the weight coefficient C(i,j) in the next row. variable j gets to "7" in S209, the CPU 1 advances the 

On the contrary, when it is determined that the current process to S211. In S211, the CPU 1 checks whether the 

variable j gets to (N-l) in S306, the CPU 1 advances the current variable i gets to "7" or not. When the variable i does 

process to S309. In S309, the CPU 1 checks whether the not yet get to "7", the CPU 1 increases the variable i in S212 

current variable i gets to (N-l) or not. When the variable i 45 and then returns the process to S206 to permutate the weight 

does not yet get to (N-l), the CPU 1 increases the variable coefficient C(ij) of the next column, 
i in S310. Then, the CPU 1 substitutes the current variable On the contrary, when it is determined that the current 

i to the following expression (6), and sets a new function variable i gets to "7" in S211, the CPU 1 advances the 

W,(x) of a variable x again. process to S213. In S213, the CPU 1 applies the two- 

50 dimensional inverse DCT to all frequency distributed picture 

FT n(2x+l) (6) data (FIG. 10(g)) including the weight coefficient C(ij) of 

w- t (x) = A — cos — — — / which the value is permutated in S208 so as to generate the 

signature picture watermark -embedded picture data of NxN 
dots shown in FIG. 10(/i). Concretely, in S213, the CPU 1 

Thereafter, the CPU 1 returns the process to S303, and 55 performs the two-dimensional inverse DCT process subrou- 

calculates the weight coefficient C(i,j) in the next column. tine shown in FIG. 8. 

On the contrary, when it is determined that the current In S401 to be the first step of this two-dimensional inverse 

variable i gets to (N-l) in S309, thc CPU 1 determines that DCT process subroutine, the CPU 1 defines functions used 

all weight coefficients in the frequency distributed picture for this process like the following expressions (7)-(10). 
data are calculated, and then terminates this subroutine so as 60 

to return the process to the main routine in FIG. 6. r-j - (7) 

In the main routine in FIG. 6, the process is advanced to w 0 (x) = A — 

S205 after finishing S204. The processes between S205 and 

S212, as shown in FIGS. 10(fc) through 10(g), are executed r— <g) 



to pick up areas indicating DC components and areas 65 w ; (jt) = A / - cos- 

indicating low frequency components (areas i-0-7 and 
j-O-7) in the frequency distributed picture data (FIG. 10(/?)), 



& 
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-continued the frequency distributed picture data of NxN pixels shown 

(9) in FIG. 11(b). Concretely, the CPU 1 calculates the two- 
dimensional DCT subroutine shown in FIG. 7. 

In the processes S504 through S510, as shown in FIGS. 

5 11(b) through 11(e), areas indicating DC components and 

, 2_ n ®y + ^ ■ ^ 10 * areas indicating low frequency components (area of i=0-7 

j{y) " V n cos IN } and j=0-7) (FIG. 11(c)) in the frequency distributed picture 

data (FIG. 11(6)) are picked up, and the signature picture 

i vam .u pni. -i ■ i* (L • ui a- .* data is extracted from those areas. 

In S402, the CPU 1 initializes the variable x indicating a . . ™ TT - . ... iL . . . .. . 

column (0 is leftmost) in the signature picture watermark- » n S504, the CPU 1 initializes the variable i indicating , a 

embedded picture data of the calculation object pixel R(x,y) f lumn of ' he P' cku P ob J ect coefficient C(ij) m the 

so as to be "0" frequency distributed picture data so as to be 0 . 

In S403, the CPU 1 initializes the variable y indicating a In S505 > lhe CPU 1 initialize the variable j indicating a 

row (0 is highest) in the signature picture watermark- row of the pickup object weight coefficient C (l j) in the 

embedded picture data of the calculation object pixel R(x,y) is frequency distributed picture data so as to be "0". 

so as to be "0". In S506, the CPU 1 reads a watermark-embedding func- 

In S404, the CPU 1 calculates the luminance value of the tion (FIG. 11(d) equal to FIG. 10(e)), and then substitutes the 
pixel R(x,y) in the signature picture watermark-embedded extraction object weight coefficient C (i j) specified by the 
picture data specified by the current variables x and y. current variables i and j i.e., evaluates the watermark- 
Concretely, the CPU 1 substitutes the current variable x to 20 embedding function at the value C(i j) so as to calculate an 
the function of the expression (8) defined in S401, and output value S'(ij) of the ith column and the jth row. 
substitutes the current variable y to the function of the In S507, the CPU 1 checks whether the current variable 
expression (10) defined in S401. Moreover, the CPU 1 j gets to "7" or not. When the variable j does not yet get to 
executes the following expression (11) based on the func- "7", the CPU 1 increases the variable j in S208, and then 
tions of the substituted expressions (8) and (10) and the 25 returns the process to S506 to calculate the output value S* 
functions of the expressions (7) and (9) defined in S401, and in the next column. 

then sets the calculated series as the luminance value of the On the contrary, when it is determined that the current 

pixel R (x,y) specified by the current variables x and y. variable j gets to "7" in S507, the CPU 1 advances the 

process to S509. In S509, the CPU 1 checks whether the 

n-i N-i 30 current variable i gets to "7" or not. When the variable i does 

y) = Yj Z C(/ ' ■ /) * w,ix) ' w j {y) not yet get to "7", the CPU 1 increases the variable i in S510, 

,vo i=0 and then returns the process to S505 to calculate the output 

value S'(i j) in the next column. 

In S405, the CPU 1 checks whether the current variable 0n the contrary, when it is determined that the current 

y gets to (N-l) or not. When the variable y does not get yet 35 variable i gets to "7" in S509, the CPU 1 advances the 

to (N-l), the CPU 1 increases the variable y in S406, and process to S511. In S511, the CPU 1 outputs output values 

then returns the process to S404 so as to calculate the (luminance values) of ixj dots calculated in S506 to the 

luminance value of the pixel R (x,y) in the next row. output unit 5. At that time, when the process object picture 

On the contrary, when it is determined that the current data is the signature picture watermark-embedded picture 

variable y gets to (N-l) in S405, the CPU 1 advances the 40 data » that picture data is the same picture data (FIG. 11(e)) 

process to S407. In S407, the CPU 1 checks whether the a » the signature picture (FIG. 10(<Q). 

current variable x gets to (N-l) or not. When the variable x According to the present invention, it is possible to 

does not yet get to (N-l), the CPU 1 increases the variable watermark identification information into an original picture 

x in S408, and then returns the process to S403 so as to data without deteriorating the picture quality of the picture 

calculate the luminance value of pixel R (x,y) in the next 45 data so as not to be recognized by a third person. Further, it 

column. is a ^ so possible to extract the watermark-embedded identi- 

On the contrary, when it is determined that the current fication information without the original picture data. Thus, 

variable x gets to (N-l) in S407, the CPU 1 determines that since il is unnecessary to keep and manage original picture 

all luminance value of pixels in the signature picture data for a right holder or the like of picture data, no mass 

watermark-embedded picture data are calculated, and then 50 storage unit is needed. 

terminates this subroutine to return the process to the main invention being thus described, it will be obvious 

routine in FIG. 6. that same may be varied in various ways. Such variations are 

In the main routine in FIG. 6, the process is advanced to not to be regarded as departing from the spirit and scope of 

S214 after finishing S213. In S214, the CPU 1 outputs the the invention, and all such modifications as would be 

signature picture watermark-embedded picture data of NxN 55 obvious for one skilled in the art are intended to be included 

pixels to the output disk unit 4. within the scope of the following claims. 

(Identification Information Extracting Process) What is claimed is: 

FIG. 9 is a flow chart illustrating an identification infer- 1. A method of watermark-embedding identification infor- 
mation extracting process in the Embodiment. mation into original picture data comprising a plurality of 

In S501 to be the first step of this identification informa- 60 P ixel values, the identification information comprising 

tion extracting process, the CPU 1 reads the watermark- numerical signals representing respective identification 

embedding function f(x) from the ROM 3. values, the method comprising: 

In S502, the CPU 1 reads the process object picture data generating combinations of mutually orthogonal basis 

of NxN pixels shown in FIG. 11(a) from the input disk unit functions for the identification values; 

2. 65 calculating corresponding weight coefficients for each of 

In S503, the CPU 1 applies the two-dimensional DCT to the combinations, each weight coefficient further cor- 

all process object picture data read in S502 so as to generate responding to a basis function of the corresponding 
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combination and being calculated based on a sum of 
products respectively corresponding to the pixels, each 
product being calculated by multiplying the pixel value 
of the pixel and a value of the basis function at a 
position of the pixel within the original picture data; 

specifying for each identification value corresponding 
input values for a watermark-embedding function 
defined as a multi-to-one function having a domain 
comprising the weight coefficients and a range com- 
prising the identification values, each specified input 
value being closest to a corresponding weight coeffi- 
cient of the identification value among plural input 
values mapped to the identification value by the multi- 
to-one function; and 

changing the weight coefficients to values equal to the 
corresponding specified input values by changing pixel 
values in the original picture data. 

2. A method of extracting identification information from 
process object picture data into which identification infor- 
mation is watermark-embedded by the watermark- 
embedding method according to the claim 1, said extracting 
method comprising: 

generating combinations of mutually orthogonal basis 
functions respectively corresponding to the identifica- 
tion values; 

calculating corresponding weight coefficients for each of 
the respective combinations, each calculated weight 
coefficient further corresponding to a basis function of 
the combination and being based on a sum of products 
respectively corresponding to pixels of the process 
object picture data, each product being calculated by 
multiplying a pixel value of the pixel and a value of the 
basis function at a corresponding pixel position within 
the original picture data; and 

calculating for each weight coefficient a value of a 
watermark-embedding function defined as a multi-to- 
one function having a domain comprising the weight 
coefficients and a range comprising the identification 
values. 

3. An extracting method according to claim 2, wherein the 
watermark-embedding function is a periodic function. 

4. An extracting method according to claim 2, wherein the 
watermark-embedding function is a continuous periodic 
function. 

5. An identification information watermark embedding 
method according to claim 2, wherein the watermark- 
embedding function transforms an interval between two 
input values into an image interval when the watermark- 
embedding function maps the two input values to a same 
output value, the image interval being narrow when the two 
input values are small and being broad when the two input 
values are large. 

6. A watermark-embedding method according to claim 1, 
wherein the watermark-embedding function is a periodic 
function. 

7. A watermark-embedding method according to claim 1, 
wherein the watermark-embedding function is a continuous 
periodic function. 

8. A watermark-embedding method according to claim 1, 
wherein the watermark-embedding function transforms an 
interval between two input values into an image interval 
when the watermark embedding function maps the two input 
values to a same output value, the image interval being 
narrow when the two input values are small and being broad 
when the two input values are large. 

9. A method of watermark-embedding identification infor- 
mation into original picture data comprising a plurality of 
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pixel values, the identification information comprising 
numerical signals representing identification values, the 
method comprising: 

applying orthogonal transformation to the plurality of 

pixel values, and generating a plurality of weight 

coefficients; 

designating weight coefficients for each of the identifica- 
tion values, the designated weight coefficients being 
selected from the plurality of weight coefficients; 

specifying for the weight coefficients of each identifica- 
tion value corresponding input values for a watermark- 
embedding function defined as a multi-to-one function 
having a domain comprising the weight coefficients and 
a range comprising the identification values, each 
specified input value being closest to the corresponding 
weight coefficient among plural input values mapped to 
the identification value by the multi-to-one function, 
and permutating the designated weight coefficients by 
the corresponding specified input values; and 

applying inverse orthogonal transformation to the plural- 
ity of weight coefficients after permutating the weight 
coefficients corresponding to all the identification val- 
ues, 

10. A method of extracting identification information 
from process object picture data to which the identification 
information is watermark-embedded by the watermark- 
embedding method according to claim 9, the extracting 
method comprising: 

applying orthogonal transformation to pixel values of the 
process object picture data and generating of a plurality 
of weight coefficients; 

selecting weight coefficients corresponding to the identi- 
fication values from the plurality of weight coefficients; 
and 

calculating for each selected weight coefficient a value of 
a corresponding watermark-embedding function 
defined as a multi-to-one function having a domain 
comprising the weight coefficients and having a range 
comprising the identification values. 

11. A method of watermark-embedding identification 
information into original picture data comprising a plurality 
of pixel values, the identification information comprising 
numerical signals representing identification values, the 
method comprising: 

applying two-dimensional discrete cosine transformation 
to the plurality of pixel values, and generating a plu- 
rality of weight coefficients; 

designating weight coefficients for each of the identifica- 
tion values, the designated weight coefficients being 
selected from the plurality of weight coefficients; 

specifying for the designated weight coefficients of each 
identification value corresponding input values for a 
watermark-embedding function defined as a multi-to- 
one function having a domain comprising the weight 
coefficients and a range comprising the identification 
values, each specified input value being closest to the 
corresponding weight coefficient among plural input 
values mapped to the identification value by the 
watermark -embedding function, and permutating the 
weight coefficients; and 

applying inverse two-dimensional discrete cosine trans- 
formation to the plurality of weight coefficients after 
permutating the weight coefficients corresponding to 
all the identification values. 

12. A method of extracting identification information 
from process object picture data into which identification 
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information is watermark-embedded by the watermark- 
embedding method according to claim 11 the extracting 
method comprising: 
applying two-dimensional discrete cosine transformation 
to pixel values of the process object picture data, and 
generating a plurality of weight coefficients; 
selecting weight coefficients corresponding to the identi- 
fication values from the plurality of weight coefficients; 
and 

calculating for each selected weight coefficient a value of 
a watermark-embedding function defined as a multi- 
to-one function having a domain comprising the weight 
coefficients and having a range comprising the identi- 
fication values. 

13. A watermark-embedding apparatus for watermark 
embedding identification information into original picture 
data comprising a plurality of pixel values, the identification 
information comprising numerical signals representing iden- 
tification values, said watermark-embedding apparatus com- 
prising: 

watermark-embedding function hold means for holding a 
watermark -embedding function defined as a multi-to- 
one function having a domain comprising a set of 
weight coefficients and having a range comprising the 
identification values; 

orthogonal transformation means for applying orthogonal 
transformation to the plurality of pixel values, and 
generating a plurality of weight coefficients comprised 
in the set of weight coefficients; 

weight coefficient permutation means for designating 
weight coefficients from the plurality of weight coef- 
ficients for each identification value, specifying for 
each designated weight coefficient a corresponding 
input value closest to the weight coefficient among 
plural input values mapped to the identification value 
by the watermark-embedding function, and permutat- 
ing each designated weight coefficient by the corre- 
sponding specified input value; and 

inverse orthogonal transformation means for applying 
inverse orthogonal transformation to the plurality of 
weight coefficients after permutation of the designated 
weight coefficients. 

14. An extracting apparatus for extracting identification 
information from process object picture data in which the 
identification information is watermark-embedded by the 
watermark-embedding apparatus according to claim 13, said 
extracting apparatus comprising: 

watermark-embedding function hold means for holding a 
watermark-embedding function defined as a multi-to- 
one function having a domain comprising a set of 
weight coefficients and a range comprising the identi- 
fication values; 

orthogonal transformation means for applying orthogonal 
transformation to pixel values of the process object 
picture data and generating therefrom a plurality of 
weight coefficients; 

means for selecting weight coefficients corresponding to 
the identification values from the plurality of weight 
coefficients; and 

calculation means for calculating a value of the 
watermark-embedding function for each of the selected 
weight coefficients. 

15. A computer readable medium storing a program to 
control a computer, said program comprising instructions 
for: 
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applying orthogonal transformation to a plurality of pixel 
values and generating a plurality of weight coefficients; 
designating weight coefficients from the plurality of 
weight coefficients for each of plural identification 
values represented by respective numerical signals; 
specifying for the designated weight coefficients of each 
identification value corresponding input values for a 
watermark-embedding function defined as a multi-to- 
one function having a domain comprising the plurality 
of weight coefficients and having a range comprising 
the identification values, and permutating the desig- 
nated weight coefficients in the plurality of weight 
coefficients by the corresponding specified input 
values, each specified input value being mapped to the 
identification value by the multi-valued function and 
being closest to the corresponding weight coefficient 
among plural input values mapped to the identification 
value by the watermark-embedding function; and 
applying inverse orthogonal transformation to the plural- 
ity of weight coefficients after permutating the desig- 
nated weight coefficients for all the identification val- 
ues. 

16. A computer readable medium storing a program to 
control a computer, said program comprising instructions 
for: 

applying orthogonal transformation to a plurality of pixel 
values of process object picture data into which iden- 
tification information is watermark-embedded, and 
generating a plurality of weight coefficients; 
selecting from the plurality of weight coefficients corre- 
sponding weight coefficients for each of plural numeri- 
cal signals representing respective specified values; and 
for each selected weight coefficient, determining a corre- 
sponding domain value of a watermark-embedding 
function defined as a multi-to-one function having a 
domain comprising the plurality of weight coefficients, 
having a range comprising the specified values, and 
mapping each selected weight coefficient to the corre- 
sponding specified value; and 
modifying the plurality of pixel values based on the 
determined domain values to output identification data. 

17. A computer readable medium storing a program to 
control a computer, said program comprising instructions 
for: 

applying orthogonal transformation to a plurality of pixel 
values of original picture data, and generating a plu- 
rality of weight coefficients; 
selecting from the plurality of weight coefficients corre- 
sponding weight coefficients for each of plural numeri- 
cal signals representing respective identification val- 
ues; and 

for each selected weight coefficient, determining a corre- 
sponding domain value of a watermark-embedding 
function defined as a multi-to-one function having a 
domain comprising the plurality of weight coefficients, 
having a range comprising the specified values, and 
mapping each selected weight coefficient to the corre- 
sponding identification value; and 
modifying the plurality of pixel values based on the 
determined domain values to output watermark- 
embedded picture data. 
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ABSTRACT 



A digital watermarking process whereby an invisible water- 
mark inserted into a host image is utilized to determine 
whether or not the image has been altered and, if so, where 
in the image such alteration occurred. The watermarking 
method includes the steps of providing a look-up table 
containing a plurality of coefficients and corresponding 
values; transforming the image into a plurality of blocks, 
wherein each block contains coefficients matching coeffi- 
cients in the look-up table; and embedding the watermark in 
the image by performing the following substeps for at least 
some of the blocks: First, a coefficient is selected for 
insertion of a marking value representative of a correspond- 
ing portion of the watermark. Next, the value of the selected 
coefficient to used to identify a corresponding value in the 
look-up table. Finally, the identified coefficient is left 
unchanged if the corresponding value is the same as the 
marking value, and is changed if the corresponding value is 
different from the marking value. After the insertion of the 
watermark, the image may be stored in a lossy-compression 
form, thus permitting efficient storage and distribution. 
Moreover, the method may be used to produce two output 
signals for authentication: (1) a meaningful pattern to facili- 
tate a quick visual check, and (2) an additional signal to 
detect unauthorized alteration. The method can be applied to 
an image compressed using JPEG or other techniques, such 
as Wavelet compression, and the marked image can be kept 
in the compressed format. Any alteration made on the 
marked image can be localized, making the method suitable 
for use in a "trustworthy" digital camera or camcorder. 

14 Claims, 13 Drawing Shcets- 
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WATERMARKING SCHEME FOR IMAGE to a human observer whereas an invisible watermark is one 

AUTHENTICATION that is not perceptible to a human but may be extracted by 

a computer or other electronic means. Whether visible or 

I. FIELD OF THE INVENTION invisible watermarking is employed depends upon the par- 

5 ticular application. The following references may be Con- 
ine present invention concerns a new method and system suhed fof background on digital wat ermarking: 

for watermarking digital images for authentication purposes. w . , . , . „ . _. . , 

& & 6 r r [1] Mintzer, et al., "Effective and Ineffective Digital 

II. BACKGROUND OF THE INVENTION Watermarks," IEEF-ICIP, 1997. 

[2] Friedman, "The Trustworthy Digital Camera: Restor- 

A black-white or grey scale image of size XxY can be 30 mg Credibility t0 tne photographic Image," IEEE Trans, on 

described by its brightness in a plane, i.e., by a function Consumer Electronics, November 1993. 

f(x,y) where f is the brightness at a point with coordinates Schneider, el aL, "A Robust Content Based Digital 

(x,y). A digital grey scale image comprises ; pixek arranged ^ for Authent ication," IEEE-ICIP, 1996 

in a raster. Such an image can be described by f(ij) where & & 

the brightness f takes on a value from a discrete set of values is W Storck "A New Approach to Integrity of Digital 

called quantized values, and (ij) is a pair of integers. An ^ages," IFIP Conf - on Moblle Communication, 1996. 

example is a digital image of 512x512 pixels and each pixel [5] Yeung, et al, "An Invisible Watermarking Technique 

takes on an integer value between 0 and 255. In this for Image Verification," IEEE-ICIP, 1997. 

specification, an image shall mean a digital image unless [6] Swanson, et aL, "Robust Data Hiding for I mages, = 

otherwise noted. 20 IEEE DSP Workshop, 1996. 

A common way to represent the pixels of a digital color [7] Koch, et al, "Towards Robust and Hidden Image 

image is to use 3 numbers denoting the red component, Copyright Labeling," IEEE Workshop on Nonlinear Signal 

green component, and blue component. In this way, the and Image Processing, 1995. 

image is represented in the R-G-B color coordinate system. rg] ^ng, et al., "On Resolving Rightful Ownership of 

Another way to describe a color pixel is to use the luminance 25 Dig ; ta i i mages by Invisible Watermarks," IEEE-ICIP, 1 997. 

and 2 chrominance components, where the luminance cor- Qne lication of di ^ tal waterraarking is in lhe field of 

responds to the brightness. There are many other color . £ r , , . ° , . . . ° . . . , 

coordinate s stems digital photography, in which images are captured with a 

^ ' digital camera or photographs are digitized. In these cases, 

Trie process of representing an image by a stream of 3Q it would be advantageous to embed an invisible watermark 

* ones' and 'zeros', i.e. using bits, is commonly referred to as m the at the ^ of captur e or digitizing. This 

image coding. That is, an image is converted to a binary watermark could be used later (e.g., in a court of law) to 

stream by image coding. Decoding refers to the process of verify that the is aume ntic, i.e., has not been altered. 

obtaining the image from the binary stream. Image com- Tt . . ^ t t tt _ . c * • i * c ,i_ 

. & - f r j - t i_ , fL '. It is important that a method or watermarking for authen- 
pression refers to the process of reducing the number of bits 

T . * . ,11 35 tication cam 
to represent a given image. In many image coding methods, 

it is desirable to use the fewest number of bits to represent 0) P ermit ™™ to determine whether an image has 

the image. For this reason, image coding and image com- been altered or not; 

pression are often used synonymously. (2) identify where in the image such alteration occurred; 

Image coding or compression can be lossless or lossy. A 40 anc * 

lossless coding method produces a binary stream from (3) allow the watermarked image stored in a lossy- 

which the original image can be obtained exactly. A lossy compression format (such as JPEG), 

coding method produces a binary stream but the decoded In addition, it is highly desirable 

image, called the compressed image, is not exactly the same (4) to integrate the watermark with the host image rather 

as the original image. In lossy coding, the compressed image 45 than as a separate data file; and 

can look indistinguishable from the original, or it can look t0 have the watermar k invisible under normal viewing 

different from the original, in which case the difference conditions. 

shows up as artifacts. Previously known methods for image authentication do 

Instead of being described by its pixel values, a digital not have all of the above capabilities. The digital signature 

image can also be described in the * frequency domain' or 50 methods (e.g., above cited references [2] [3] [4]) do not have 

more generally the 'transform domain'. The NxM values of capabilities 2 and 3; the pixel-domain watermarking meth- 

f is transformed to a set of numbers called transform 0 ds (e.g., above cited reference [5]) do not have capability 

coefficients, usually also NxM in number. Commonly used 3 ; t h e frequency-domain data hiding schemes (e.g., above 

transformations include the Discrete Cosine Transform cited [6] [7]) cannot always localize alterations and may 

(DCT), Discrete Fourier Transform (DFT), and Wavelet 55 introduce excessive distortion. Since the present invention 

Transform. and the pixel-domain method [5] have certain similarity, that 

Current computer and information technology allow easy method will be briefly reviewed and the difference pointed 

editing and perfect reproduction of digital images, which in out. 

turn can lead to problems with copyright protection, own- The method presented in [5] embeds a watermark, which 

ership verification and authentication. Such problems are 60 is a binary pattern, in the pixel domain, using a look-up table 

addressed by digital watermarking, which concerns pro- (LUT). The method is illustrated in FIG. 1, where the 

cesses that embed or insert data into a multimedia data (unmarked) grey scale image consists of a block of 8x8 

object. The inserted data are often called digital watermarks. pixels whose values are shown and the pattern to be embed- 

Depending on the application, digital watermarking may be ded is the letter "I", also formed with a block of 8x8 pixels, 

applied to different types of data, including digital still 65 Suppose the black pixels of the pattern correspond to "0" 

images, digital audio and digital video. For images, a visible and the white pixels correspond to "1". These "l"s and "0"s 

watermark is one that is intentionally made to be noticeable are called marking values. As shown in FIG. 1, the top row 
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of the LUT are the luminance values of the unmarked image image, into which we embed a watermark of the pattern 

and the bottom row are binary, i.e., "1" or "0". The 4th "PUEE" (shown in FIG. 2B) using the method of the present 

number in the first row of the image has a value of 153 as invention. The watermarked image, shown in FIG, 2B, is 

shown; the binary pattern corresponding to this pixel is indistinguishable from the unmarked original, FIG. 2 A. Two 

black and therefore has the value "0". From the LUT, 153 5 modifications are then made of the image. "Princeton Uni- 

corresponds to a "0", agreeing with the pixel value of the versity" on the top right corner is changed to "Alexander 

pattern. So the number 153 is unchanged in the marked Hall" and "Copyright 1997" in the lower left corner is 

image, i.e., the 4th pixel in the first row of the marked image changed to "January 1998", as illustrated in FIG. 2C. Also 

has the value 153. The first number in the second row of the shown in FIG. 2C is that the watermark is extracted from this 

image has the value 144. The corresponding binary pattern 10 modified image, which clearly shows where modifications 

is white and therefore has the value "1". But from the LUT, have taken place. 
144 corresponds to a "0". So the number 144 is changed to 

143 for which the corresponding value in the table is "1 ". ni - SUMMARY OF THE INVENTION 
Thus, the first pixel in the second row of the marked image a primary object of the present invention is to provide a 
has the value 143. All pixels in the original image are 15 digital watermarking process whereby an invisible water- 
processed in this manner. That is, if the luminance of a pixel mark inserted into a host image can be utilized to determine 
in the original image does not map to the value in the whether or not the image has been altered and, if so, where 
corresponding binary pattern by the LUT, the luminance m the image such alteration occurred. This and other objects 
value is changed to a new value which is close to the original Q f the invention are achieved by the methods disclosed 
value and which corresponds to a binary value that agrees 20 herein for applying a digital watermark to an image, 
with the binary pattern. jne mve ntive methods include the step of deriving from 

The marked image is made up of an 8x8 block of pixels the image a p i ura lity of component images, wherein each 

from which the watermark is easily extracted by referring to comp onent image contains coefficients. The inventive meth- 

the LUT The fourth pixel in the first row of the marked ods employ at least one look-up table containing a plurality 

image is 153, for which the corresponding binary value from 25 of coefficients and corresponding values, wherein at least 

the LUT is "0". So the fourth pixel in the first row of the of the look . up table coefficients match coefficients in 

extracted pattern is black. Similarly, the first pixel in the the component images. Awatermark is then embedded in the 

second row of the marked image is 143, for which the imag6i ^ last stepj embedding the watermark, is carried 

corresponding binary value from the LUT is " 1". So the first out 5 y performing the following steps for at least some of the 

pixel in the second row of the extracted pattern is white. 30 comp0 nent images: (1) selecting a coefficient into which a 

When all the pixels of the marked image have been pro- mar king value, representative of a corresponding portion of 

cessed in this manner, a pattern of "I" will have been the watermark, is to be embedded; (2) using the value of the 

extracted, selected coefficient to identify a corresponding value in the 

If a pixel in the marked image is changed, the changed look-up table(s); (3) leaving the identified coefficient 

value will be mapped to either "1" or "0", each with a 35 unchange d if the corresponding value is the same as the 

probability of 0.5, So there is a 0.5 probability that the marking value; and (4) changing the identified coefficient if 

extracted watermark for that pixel will be different from the the corresponding value is different from the marking value. 

corresponding pixel in the original pattern. Such a possible of a ^ implementalion of the 

change in a single pixel may or may not be observable by a ig tfa ^ (he embeddi st the identified 

viewer. However, if a group of neighboring pixels is 40 u ^ •• « * i u • 

, , , ' , ... f \ & ,. & r „ , coefficient is changed minimally to a value having a corre- 

changed, the probability that the corresponding part of the i ■ fu i i ♦ ui u* u • *u *u 

4 * j v i -ii i i j-n? * • * -a .i sponding value m the look-up table which is the same as the 

extracted watermark will look different is significantly i • i r i *u m * c j «= • * • 

, . ^i-i j marking value. For example, the identified coefficient is 

increased. Images are often lossily compressed to save ferabl ch d minimall b chan ^ its va , ue t0 , ha , 

transmission time and storage If an mage is watermarked of ^ neafest coefficien , havin a co^pond^ look . up 

using the method just described, and if the marked unage is 45 uble ya , ue ^ (0 ^ markin ya]ue 

then lossily compressed, the watermark inserted in the „ ^ _ , , 

image will be changed due to the compression process. u ™ an ?, lh l er as Pf c ! of the P rei f Dl 1 ? ve J n J UoD C0DC o ern f 

Therefore, the watermarked image from such an approach * hu 5 in S the efficients prior to the embedding step. Such 

cannot be lossily compressed without adversely affecting the involve concatenating the coefficients of a 

watermark. Another disadvantage of the above method is 50 P 1 "" 1 ^ f blocks mto a string and randomly shuffling the 

due to the fact that a human viewer can easily notice the orderof the coefficient in the stnng In this embodiment, the 

changes in the pixel values in the smooth regions of the stno & 15 uns ^fHed after the embedding step, 

image due to the embedding process, making it difficult to Preferably, for error correction purposes, the marking 

insert the watermark in the smooth regions of an image. value ( e -g-» marking bit) is embedded multiple times in each 

As described in greater detail below, the present invention 55 block > and a majority voting process is employed to decode 

inserts a watermark in the quantized transform domain tDe respective blocks. 

coefficients using a lookup table. This allows the water- The preferred embodiments also include the step of 

marked image to be stored in compressed form. The present identifying selected coefficients as being unembeddable, 

invention can also insert the pattern in the smooth regions of such that the unembeddable coefficients are not employed to 

an image and can embed content based features in addition 60 embed marking values. For example, DC coefficients (i.e., 

to a pattern, so the reliability of detecting alterations of the Cqq) of the DCT process may be considered unembeddable. 

image is significantly increased. In addition, a shuffling In addition, a threshold value may be selected and used such 

scheme of the present invention can also be applied to that coefficients having a value below the threshold value are 

embedding a watermark in the pixel domain of smooth considered small valued and thus unembeddable. 

regions. 65 The preferred embodiments of the present invention can 

The result achieved by present invention is illustrated determine whether a marked image has been altered, and 

with FIGS. 2A, 2B and 2C. FIG. 2A is a JPEG compressed where such modification took place. In addition, after the 
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insertion of the watermark, the image may be stored in a The JPEG method is illustrated in FIG. 3. To compress an 

lossy-compression form, thus permitting efficient storage image using the method, the image is first divided into 

and distribution. Moreover, the preferred embodiments pro- blocks, typically of 8x8 pixels. Each block of pixels may be 

duce two output signals for authentication: (1) a meaningful viewed as a component of the original image and hence be 

pattern to facilitate a quick visual check, and (2) an addi- 5 called a component image. For each block, a two- 

tional signal to detect unauthorized alteration. In addition, dimensional discrete-cosine-transform (DCT) is performed, 

other information, such as content features, can be embed- resulting in 8x8 DCT coefficients for that block. The coef- 

ded into the image. The invention can be applied to an image ficients represent the spatial frequency components of the 

compressed using JPEG or other techniques, such as Wave- 8x8 block of pixels and are denoted by c, y , where i and j 

let compression, and the marked image can be kept in the no range from 0 to 7. The c^ coefficient is called the DC 

compressed format. Any alteration made on the marked coefficient, corresponding to the DC component of the 

image can be localized, making the preferred embodiment original image block in both horizontal and vertical 

suitable for use in a "trustworthy" digital camera or cam- directions, while the Cj 7 coefficient contains the highest 

corder. Furthermore, since the invention is computationally frequency content of that block. 

and structurally efficient (including the use of a look-up is To lossily compress the image, each DCT coefficient is 

table), it can be applied in digital video authentication. quantized, and the normalized quantized coefficients are 

Other features and advantages of the present invention are run-length encoded, followed by Huffman or arithmetic 

described below encoding to produce a bit stream. To obtain the recon- 
structed image, the above process is reversed. That is, the 

IV BRIEF DESCRIPTION OF THE DRAWINGS 2 0 normalized quantized coefficients are recovered from the bit 

- , , A , . iLJ r stream, and inverse DCT of these coefficients taken to 

FIG. 1 depicts a pixel domain watermarking method or , ' 0 . . A11 , . . r 0 0 . t , 

Al _ , •: j r rr-i produce 8x8 pixels. All blocks of the 8x8 pixels thus 

the above cited reference [51. r , . . , . . . c t *\ . ir 

L J obtained are put together to form the reconstructed image. If 

FIGS. 2A-2C illustrate the overall result achieved by the tfae q Uantized DCT coefficients differ very little from the 

present invention. ^ or jginaI unquantized coefficients, then the reconstructed 

FIG. 3 depicts a JPEG still image compression method. image differ imperceptibly from the original image. 

FIG. 4 illustrates an image and its wavelet coefficients. Otherwise, the reconstructed image will differ noticeably 

FIG. 5 is a block diagram of a watermarking process of from the original image, 

the present invention. Many JPEG encoders use a quantization table to quantize 

FIG. 6 depicts a transform domain embedding process 30 the DCT coefficients by dividing each coefficient by the 

employed by the present invention. corresponding entry in the table and then rounding it to the 

FIGS. 7A and 7B are block diagrams illustrating the near ^ st . inte f r w u hich ^ <? Ued J he ^"nalized quantized 

embedding and authentication procedures, respectively, in coefficient. To achieve different level of compression the 

accordance with the present invention. cntacsof a table is scaled up or down by a number called the 

™^ « * 1 . ., - a , ™ • , is quality factor Q. Table 1 together with a value of Q between 

FIGS. 8A and 8B are similar to FIGS. 7A and 7B in that ? J* ^ ^ for ^ ^ tioa> „ foUows . For 

they illustrate embodiments of the watermark embedding Q=50 ^ ^ ^ ux<J UQCD ^ d „ the quantizatiori table . 

and authentication procedures, respectively, but include ^ . each DCr ^ mulli Ued b the corre . 

shuffling and inverse shuffling steps in accordance with di of TaWe j ^ ^ ^ ^ founded and 

another embodiment of the invention. , r, ~ cn „ e .* „ j 

vm^umivu ^ clipped. For Q>50, a new table for quantization is generated 

FIG. 9 depicts the shuffling process. by multiplying Table 1 by (100-Q)/50. For Q<50, the new 

FIG. 10 illustrates a backup embedding scheme that may quantization table is generated by multiplying Table 1 by 

be employed in connection with smooth regions. 50/Q. It should be mentioned that the present invention does 

FIG. 11 illustrates three types of frames (I, P, B) in a video not depend on this or any other quantization table, 

signal. 45 

FIG. 12 illustrates a step in MPEG coding using motion TABLE 1 



compensation. 

V. DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 
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We first illustrate the present invention by showing how 
to insert a binary pattern as the watermark in a JPEG 
compressed grey-scale image. The present invention, 
however, may be extended to other compression methods, 
such as Wavelet compression, to color images, and to video. 55 
Before describing the preferred embodiments of the 

invention, we will provide a brief overview of the JPEG V.2 Subband and Wavelet Image Compression 

method for lossy compression. Details can be found in the Another well known image compression method is based 

paper G. K. Wallace, "The JPEG still picture compression on subbands. In this method, an image is filtered and the 

standard," IEEE Trans on Consumer Electronics, February 60 outputs of the filters are sub-sampled or down sampled. (To 

1992. sub-sample by a factor of 2, one out of every two samples 

V.l Overview of JPEG Lossy Compression are deleted. To sub-sample by a factor of K, all but one from 

The JPEG method for image compression has been every K samples are deleted.) The results are component 

adopted as an international standard. When the image is images, or more specifically highpass and lowpass compo- 

lossily compressed, the compressed image may differ imper- 65 nent images if the filters used are highpass and lowpass 

ceptibly or significantly from the original image, depending filters. The component images usually bear certain resem- 

on the parameters chosen to perform the compression. blance to the original image. The process of filtering and 
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sub -sampling is often repeated several times. The final V.4 Insertion of Binary Pattern Using a Look-up Table 

results are called subband coefficients, or subband images. We now discuss details of how to insert the watermark. As 

These subband coefficients are then quantized and the quan- shown in FIG. 6, a look-up table (LUT) 60, which is 

tized coefficients are then assigned to codes. In subband generated beforehand, maps every possible value of JPEG 

coding, the quantized coefficients from each subband are 5 coefficient to "1" or "0" with the lengths of consecutive "1" 

usually coded independently. Another well known method or "°" limited. Thus, the LUT 60 includes a row 60a of 

for image coding is based on wavelets, which is quite similar JPEG coefficients and a row 606 of corresponding "l"s and 

to subband coding in the initial steps. That is, the image is "°" s - To embed a " r in one of tne original JPEG coefficients 

passed through a number of filters, and the outputs of these 35 > that coefficient is unchanged if the entry of the table 

filters are subsampled to produce a number of coefficients, 10 corresponding to that coefficient is also a "1". On the other 

called wavelet coefficients. hand ' if the ori gi nal coefficient corresponds to "0" in the 

Shown in FIG, 4 is an image and its wavelet coefficients. table » that coefficient value is changed minimally to a 

Notice the varying degree of resemblance of the seven neighboring value that has a "1" entry. The procedure for 

components to the original image. The wavelet coefficients embedding of a "0" is similar, i.e., the coefficient is 

are then quantized and coded. Notice also that the seven 15 unchanged if its corresponding LUT entry is "0" but is 

components are not of the same size. The wavelet coeffi- changed minimally if its corresponding LUT entry is "1". 

cients from one filter output is often correlated with those These " rs " ^ "°' s " are ca^d marking values. It should 

from other filter outputs, and wavelet coding takes advan- be P 0 "^ out that the expression (to embed) as used herein 

tage of such information. Although in principle this advan- does not necessarily require that the original data value be 

tage can also be used in subband coding, what most 20 altered, but only that it is altered as necessary to make it 

researchers refer to as subband coding actually code each ec l uaI t0 a value wnose corresponding LUT value ("0" or 

subband separately, as mentioned in the previous paragraph. " r ) fa e 9 ual t0 the value 10 be embedded. 

Details of these image coding methods can be found, for In the example of FIG. 6, zeros ("0"s) are to be embedded 

example, in M. Vetterli and J. Kovacevic, Wavelets and in two DCT coefficients of values "-73" and "24" respec- 

Subband Coding, Prentice Hall, 1995. 25 tivel V- since tne binar V entr y corresponding to "-73" is 1, 

V.3 Overview of Preferred Embodiments "" 73 " * changed to "-74" for which the binary entry is "0" 

We first describe the present invention by demonstrating whicD is to be embedded. The coefficient value of "24" is 

how to insert a pattern in a JPEG compressed grey-scale unchanged because its binary entry is already "0". 

image. For simplicity, we shall use for illustration a binary ^ len S th of successive zeros and ones is limited to avoid 

pattern. We will discuss later how other information, such as 30 noticeable distortion. The following procedures can be used 

content features, can be embedded into the image. The t0 generate an L-entry look-up table T{.] with maximum 

embedding of a binary pattern is done by modifying some of allowed run of r. 

the normalized quantized DCT coefficients using the binary Step-1: 

pattern and a look-up table (LUT) in such a way that the Step-2: If i>r and T[i-l]-T[i-2]= . . . -T[i-r], then 

resulting marked image is visually not different from the 35 T[i]ol-T[i-l]. 

original compressed image. The inserted pattern can be Otherwise, randomly select a number from {0, 1} and set 

extracted from the marked image. The binary pattern can be T[i] to this value. 

a visually meaningful pattern such as a logo, letters, etc. Step-3: Increase iby 1. If i>L, stop. Otherwise go back to 

Note that the marked image is in a compressed form. If the Step-2. 

marked image is unchanged, then the extracted pattern will 40 Random numbers can be generated by pseudo- random gen- 
be identical to the original inserted pattern. If, on the other era tor; details can be found, for example, in R. Sedgewick, 
hand, the marked image was altered, then the extracted Algorithms in C, Addison-Wesley, 1990. 
pattern will reveal where such alteration took place, as It should be noted, however, that there are many ways to 
demonstrated earlier in FIG. 2. generate such a table and the present invention is not limited 
FIG. 5 schematically depicts a preferred embodiment of 45 to any one procedure. It should also be noted that more than 
such a watermarking process in accordance with the present one look-up table can be used to insert data and that the 
invention. As shown, an original image 10 undergoes a block second entry of the table is not limited to be binary, 
by block DCT 20 and quantization 30, which yields a set of V.5 Selection of DCT Coefficients for Data Insertion 
normalized quantized coefficients 35. The embedding pro- When the quantized coefficients are changed too much, 
cess 40 modifies the quantization coefficients 35 to give a set 50 the marked image will be visually different from the original 
of marked coefficients 45. The embedding process 40 image. On the other hand, a large change of the coefficients 
employs a prescribed set of data to be embedded 49 (i.e., the allows more information to be embedded into the image, 
watermark) and a look-up table 60, as described in detail The human visual system, in particular the just noticeable 
below, to generate the marked coefficients 45. The marked difference, has been used to determine how much can an 
coefficients 45 are then input to an encoding process 70, 55 image be changed without causing noticeable differences, 
which yields a bit stream of the compressed watermarked The human visual system is explained, for example, in A. K. 
image 80. The inserted data can be extracted by simply Jain, Fundamentals of Digital Image Processing, Prentice- 
reading from the LUT the value corresponding to the DCT Hall, 1989. The just noticeable differences is presented, for 
coefficient of the marked image. If a part of the watermarked example, in A. J. Ahumada, et. al, "Luminance-model-based 
image is altered, the DCT coefficients from that part of 60 DCT quantization for color image compression," Proc. 
image will be changed, hence the extracted watermark will SPIE, Vol. 1666, 1992. 

differ from what was originally inserted, thus identifying In accordance with the presently preferred implementa- 

where in the image the modification has occurred. This tion of the invention, several steps are taken to ensure that 

assumes that one bit of a binary watermark can be embedded the markings are invisible. As mentioned above, the lengths 

in the corresponding block of the image. The question of 65 of runs of "1" and "0" entries in the LUT is limited to avoid 

how to make sure that it is possible to embed one bit in eacb excessive modification on the coefficients. The DC coeffi - 

block is discussed below. cients is unchanged unless the quantization step is small. 
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Also, the small valued AC coefficients are unchanged. Those 
DCT coefficients that cannot be changed because of these 
constraints are called unembeddable, and those coefficients 
that can be changed are called embeddable. On the average, 
between 6 to 10 of the coefficients in a block of a typical 5 
natural image are embeddable. This means one can normally 
embed more than one bit of information in each block. All 
the embeddable coefficients in a block can be used to embed 
the same bit in the binary pattern in the manner described in 
the previous section. In this case, the extraction of the 10 
embedded information can be done using majority voting. 
The reason for using more than one coefficients to embed 
same information is explained in the next section. However, 
the number of embeddable coefficients vary significantly 
from block to block and some blocks may not contain any is 
embeddable coefficients. In particular, it is difficult to embed 
data in smooth blocks where all AC coefficients are very 
small and hence unembeddable. If no bits of the watermark 
is embedded in a block, then the alteration of that block 
would be difficult to detect. To over come this difficulty, two 20 
methods are presented later in the section entitled Smooth 
Region Embedding. 
V.6 Embed Other Data 

The watermark that was inserted in the previous discus- 
sion is a binary pattern which often can provide a quick 25 
visual way of detecting modifications. It is clear that a 
non-binary pattern can be similarly embedded. It is also 
possible to embed 'content features'. A simple example of 
such features is the most significant bit of average intensity 
of a block or of a group of blocks such as a macroblock 30 
consisting of 4 blocks (16x16 pixels). The combination of 
patterns and features is suitable for such applications as 
image authentication for "trustworthy" digital cameras. The 
embedding and authentication procedures are illustrated in 
FIGS. 7A and 7B. 35 

As shown in FIG. 7 A, the embedding process is the same 
as the basic process illustrated in FIG. 5 except that content 
features are extracted from the original image 10 or from the 
DCT coefficients 25 or from both. The content features 90 
are input to the embedding process 40 along with the pattern 40 
50. 

The authentication process, as shown in FIG. 7B, starts 
with a test image 100 (i.e., a watermarked image whose 
authenticity is to be tested). It is transformed by a block DCT 
process 20 and then quantized in the same manner as in the 45 
embedding/watermarking process to produce normalized 
quantized coefficients 36. Content features 91 are also 
extracted from the test image. The watermark is extracted by 
an extraction process 110, which is the inverse of the 
embedding process 40. In the extraction process 110, the 50 
normalized quantized DCT coefficients 36 are examined in 
view of the look-up table 60 (the same table as used in the 
watermarking process of FIGS. 5 and 7A). The outputs of 
the extract process 110 are a pattern 51 and the features 92. 
The pattern 51 can be visualized by using visualize process 55 
120, such as a printer or computer monitor, to allow a human 
observer to quickly identify which part of the image has 
been changed. The extracted pattern 51 can also be com- 
pared with the original pattern 50, if available, to detect 
possible alterations. The compare process 130 compares the 60 
extracted features 92 with the features 91 to provide an 
additional detection of possible alterations. If the test image 
100 is the same as the marked image without modification 
and if the bit stream of the test image is used for 
authentication, its normalized quantized coefficients are 65 
available from the bit stream and are the same as the 
coefficients 35, and the features 90, 91 and 92 are identical. 
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If the test image 100 is the same as the marked image 
without modification but it is presented for authentication as 
an image, then a small number of coefficients 36 may be 
different from coefficients 35 because of the rounding error 
in the computation of coefficients. Error correction coding, 
such as using majority voting as mentioned in the previous 
section, can be incorporated in the embed process 40 to 
guard against such errors. 
V.7 Smooth Region Embedding 

The percentage of smooth regions in an image varies 
widely, depending on the subject matter. But typically, 20% 
of a natural image may be regarded as smooth. As mentioned 
previously, blocks in the smooth region of an image may not 
have any DCT coefficient to insert the watermark or to 
embed features. This is true in spite of the fact that between 
10 to 15% of the DCT coefficients in a natural image are 
embeddable. Thus, on the average, there are 7 to 10 
embeddable coefficients per block on average. In other 
words, there are plenty of embeddable coefficients in an 
image, but the uneven distribution of them leaves some 
blocks with no coefficients in which to embed data. 

This difficulty can be circumvented by the following 
shuffling process. For an image divided into blocks of 8x8 
pixels, the quantized DCT coefficients of all blocks can be 
concatenated into a single string, of which the first 64 
numbers are from the first image block, the next 64 numbers 
are from the second image block, and so forth. Some of the 
groups of 64 coefficients may not have any coefficient that 
are embeddable. Suppose the order of this string is randomly 
shuffled to produce a second string. Each coefficient in the 
second string comes from one and only one coefficient in the 
first stream, that is, before shuffle. Because of the random 
shuffle, the probability that each group of 64 numbers in the 
second string contains at least one embeddable coefficient is 
significantly increased. In fact, it can be calculated that the 
probability of no embeddable coefficients in a group of 64 
coefficients in the second string is about 0.1%. Therefore, 
the coefficients in the shuffled string can be used to embed 
data as described previously, even though these 64 coeffi- 
cients are not those computed from that block. 

The embedding process when shuffling is used is illus- 
trated in FIG. 8 A. It is identical to that described above 
except that the quantized DCT coefficients of all blocks are 
concatenated into a single string and randomly shuffled, in 
the process 140, before embedding. Suppose the blocks arc 
of size 8x8. Then each of the 64 coefficients of the shuffled 
string are used to embed the data for one block. 

This procedure is illustrated with the help of FIG. 9 for the 
case of embedding a binary pattern. Suppose the coefficients 
of the first block have values: 171 165 144 ... p the 
coefficients for the second block have values: 180 201 
192 . . . , the coefficients of the third block have values: 130 
125 128 ... , etc., as shown in the first sequence. The 
shuffling reordered this sequence and produces the second 
sequence. In this illustration, the first block of the shuffled 
coefficients have values: 180 165 125 130 ... , where the 
first coefficient "180" comes from the first coefficient of 
block 2 in the original (unshuffled) sequence, the second 
coefficient "165" is not moved by the shuffle, the third 
coefficient "125" comes from the second coefficient of the 
original block 3, and the fourth coefficient "130" comes 
from the first coefficient of the original block 3. Suppose a 
"1" is to be embedded in block one and suppose this 
embedding changes the second coefficient "165" to "164" as 
shown in the third sequence of FIG. 9. After embedding, all 
coefficients are inverse shuffled, which is the fourth 
sequence of FIG. 9. It should be clear that the embedding of 
features can be done identically. 
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FIG. 8B depicts the authentication process for a test image 
100. It is transformed by a block DCT process 20, quantized 
by the process 30, and then shuffled by the process 140, in 
the same manner as in the embedding process. The water- 
mark is extracted by an extraction process 110. Using the 
LUT 60, the extract process 111 extracts the embedded 
pattern 53 and the embedded features 93. The extracted 
pattern 53 is visualized by the process 122 and the features 
93 are compared with features 91 by the compare process 
132. The visualize process 122 may be different from the 
visualize process 120 (FIG. 7) and the compare process 132 
may be different from the compare process 130 (FIG. 7) 
because processes 122 and 132 may include inverse shuffle 
and other operations. 

It should be noted that in addition to the random shuffling 
involving all the coefficients in one shuffle, other reordering 
of the DCT coefficients is possible, such as dividing all the 
coefficients of the image into several parts and shuffling each 
part separately. 

It is mentioned above that the pixel domain embedding 
process may encounter difficulty when embedding data in 
the smoothed region of the image. The shuffling scheme 
described can be used to embed data in the pixel domain in 
the smooth regions of the image. As a simple example, 
suppose we embed one bit in every four pixels. Each pixel 
in the image is determined to be in a smooth region or not. 
Those in the smooth region are labeled unembeddable, and 
those not in the smooth region are embeddable. All pixels 
are then randomly shuffled. With the shuffle, the probability 
that any four pixels will have at least one embeddable pixel 
is significantly increased. The data is embedded and the 
pixels are inverse shuffled after the embedding. 

Another way of smooth region embedding is to embed 
data in blocks that are not smooth but whose location bears 
a fixed relationship to that of the smooth block in question. 
This method is simple to implement, as illustrated in FIG. 
10. As shown, instead of embedding one bit independently 
in each block (8x8 pixels), we take a macroblock (16x16 
pixels) as a unit, and use two bits to embed data correspond- 
ing to the current macroblock which is not smooth, and use 
the other two bits to embed the data corresponding to the 
companion macroblock which is smooth. That is, we use 
companion macroblocks as a backup. In the illustration of 
FIG. 10, the companion macroblocks are separated by half 
the image height, i.e., the two companion macroblocks are 
at location (i,j) and (ij+H/2), where H is the image height in 
terms of number of blocks. One can also group 4 macrob- 
locks located, say at (ij), (ij+H/2), (i+W/2j) and (i+W/2, 
j+H/2) as companion macroblocks, where W is the image 
width in terms of number of blocks. 
V.8 Detection of Alteration 

As described above, modifications on a watermarked 
image are detected by visualizing the extracted pattern, or by 
comparing the pattern with the original pattern, or by 
comparing the extracted features using the LUT with the 
features of the test image. If the watermarked image has 
been changed, then the DCT coefficients from that part of the 
image that have been modified will be changed, hence the 
extracted watermark will be changed. Suppose one bit from 
the watermark is inserted into each block. The watermark 
extracted from the block in question after alteration has a 
50% probability to be a "1" or a "0", hence there is 50% 
probability that the extracted bit may disagree with the bit 
originally inserted before the alteration. Since intentional 
modification of the image usually involves regions consist- 
ing of more than one block, this 50% detection probability 
is increased to 75% if two blocks are changed, to 87.5% if 
three blocks are changed, and to 93.75% if four blocks are 
changed. 



15 



Furthermore, suppose a block has more than one 
embeddable coefficients and we embed the same bit in all 
these coefficients. Suppose this block is modified and there 
are now N embeddable coefficients after modification. The 

5 probability that the extracted watermark bit from each 
coefficient is "1" or "0" each with a probability of 0.5. So the 
probability of all extracted bits agreeing with the originally 
inserted bit is reduced to (0.5) N. In general, the probability 
of correct detection of image modification will be increased 

1(J if more than one bit per block are inserted into the marked 
image. 

If shuffling is used, the detection of alteration can be 
illustrated by the following simple example. Suppose one 
block was altered. The embeddable coefficients from this 
block are now in different shuffled blocks, and the extracted 
bits from each of these coefficients have 0.5 probability of 
being changed. Suppose this is the only block that has been 
altered. Then error correction coding from the bits extracted 
from the shuffled blocks will identify those coefficients that 
2Q have been changed. These coefficients would have to come 
from the altered block. The 'content features' embedded as 
watermarks can also be used to detect alteration. 

V. 9 Deterring Forgery of Watermarks 

If the same look-up-table and the same pattern are used to 
25 mark many images, it is possible to obtain information about 
the table and the pattern from these images. Thus, to deter 
forgery, it is important to deter any attempt obtain informa- 
tion about the table and the pattern from the images. A 
simple way do this is to modulate the bits to be inserted by 
30 a pseudo random sequence. Depending on the computation 
and memory resources available for the particular 
application, such a pseudo random sequence can be either a 
fixed one or one selected from a list of such sequences. The 
shuffling process as described previously also helps to deter 
35 such an attempt, because it is difficult to forge watermarks 
without prior knowledge of the shuffling. 

VI. Extensions 
VT.l Double Watermarking for Authentication and Owner- 
ship Verification 

40 Previous work on double watermarking mainly empha- 
sized embedding multiple labels (such as an ownership 
label, recipient label, etc.) using the same embedding 
approach. The present invention may be used in combination 
with watermarking methods for ownership verification such 

45 as described in the above cited reference [8]. 

VI. 2 Color Images 

Whereas each pixel in a grey scale image is represented 
by a single number, each pixel in a color image is repre- 
sented by more than one numbers, such as the three com- 

50 ponents in the RGB or the YCrCb color coordinates. The 
present invention can be applied separately to each of the 
coordinates or to combination of coordinates. 
VI.3 Other Compression Methods 

The present invention can be applied to images com- 

55 pressed using methods other than JPEG. In wavelet 
compression, for example, the image is transformed into a 
number of component images from which the original image 
can be reconstructed. The component images are quantized. 
The present invention can be used to insert a watermark or 

60 watermarks into the component images by modifying the 
quantized coefficients using one or more look-up tables. 
VI.4 Multi-level Data Embedding and Unequal Error Pro- 
tection 

As mentioned above, two sets of data, namely, a mean- 
65 ingful visual pattern and a set of low-level content features, 
may be embedded in the image for authentication purposes. 
Multilevel data embedding and unequal error protection can 
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be employed to embed several sets of data with different 
error protection capabilities in accordance with their impor- 
tance. 

VI .5 Video Coding and Authentication of Video 

A video consists of many images arranged in an orderly 
sequence. The individual images are called frames or pic- 
tures. Some video systems also divide each frame into two 
fields, called the even field and the odd field. Digital video 
refers to a video sequence in which the individual frames 
have been digitized. 

Video coding or video compression relies on the infor- 
mation redundancies found within each frame and among 
frames. A simple way to code a digital video is to code each 
frame separately, such as in the method known as motion 
JPEG. In the well known methods of MPEG-1 and MPEG-2, 
the frames are divided into three types: the intra-coded or 
I -frames, the predictive-coded or P-frames, and the 
bi-directional coded or B-frames. A typical MPEG coded 
video may have a pattern such asIBBPBBPB...,as 
illustrated in FIG. 11. The I-frames are codes by themselves, 
similar to JPEG coding. Each P-frame is divided into blocks, 
typically of size 8x8 pixels. Each block of pixels is matched 
with the pixels in the previous I or previous P-frame and the 
best 8x8 pixels in that I- or P-frame is called the reference 
block. A block in a P-frame and its reference block in a 
previous I- or P-frame is illustrated in FIG. 12. The differ- 
ence is taken between each block and the reference block. 
This difference, called the residue or the displaced-frame- 
difference, is coded using a JPEG like method. The relative 
position of the block in the current P-frame and that of the 
reference block is called the motion vector of the block. 
Motion vectors are often determined using 4 blocks together 
(16x16 pixels), called a macroblock. Motion compensation 
coding refers to the determination of motion vectors and the 
coding of the residues. 

The coding of B-frames is identical to that of P-frames 
except each block in the B -frame in question is matched with 
the I-frames and P frames either in the backward or the 
forward directions, or both. As for the P-frames, the residue 
after motion compensation of the B-frames is coded. 

The present invention can be applied to digital video 
authentication. Consider, for the purpose of discussion, 
digital video compressed using the MPEG-1 and MPEG-2 
standards. Since the I-frame is compressed in much the same 
way as a JPEG still image, the present invention can be 
applied straightforwardly. 

To watermark P-frames that are coded using motion 
compensation, one can begin with a digital video where each 
frame has been watermarked using the approach described 
above. After MPEG compression, the watermark embedded 
in the I-frames can be extracted as discussed above. The 
extracted watermark embedded into the P-frames before 
MPEG encoding may be distorted from that originally 
inserted. If this is the case, then the residue after motion 
compensation, called the displaced frame difference, is 
modified. This process is repeated until the embedded water- 
mark can be reliably extracted. 

The true scope of the present invention is not limited to 
the presently preferred embodiments disclosed above. For 
example, the invention is not limited to processes employing 
the DCT, or to processes in which 8x8 or even 16x16 blocks 
of data are used to embed a "0" or "1 " bit of a watermark. 
Other possible modifications of the preferred embodiments 
will be apparent to those skilled in the art. 

We claim: 

1. A method for applying a digital watermark to an image, 
comprising the steps of: 



10 



15 



20 



25 



30 



35 



40 



45 



50 



55 



60 



65 



(A) deriving from the image a plurality of component 
images, wherein each component image contains coef- 
ficients; 

(B) providing at least one look-up table containing a 
plurality of coefficients and corresponding values, 
wherein at least some of the look-up table coefficients 
match coefficients in said component images; 

(C) embedding said watermark in said image by perform- 
ing the following substeps for at least some of said 
component images: (1) selecting a coefficient into 
which a marking value, representative of a correspond- 
ing portion of the watermark, is to be embedded; (2) 
using the value of the selected coefficient to identify a 
corresponding value in said look-up table(s); (3) leav- 
ing the selected coefficient unchanged if the corre- 
sponding value is the same as the marking value; and 
(4) changing the selected coefficient if the correspond- 
ing value is different from the marking value; 

further comprising the step of identifying selected coef- 
ficients as being unembeddable, wherein said unem- 
beddable coefficients are not employed in step (C) to 
embed marking values, and wherein some of said 
coefficients are identified as DC coefficients, and said 
DC coefficients are considered unembeddable; and 

further comprising selecting a threshold value, wherein 
coefficients having a value below said threshold value 
are considered small valued and thus unembeddable. 

2. A method for applying a digital watermark to an image, 
comprising the steps of: 

(A) deriving from the image a plurality of component 
images, wherein each component image contains coef- 
ficients; 

(B) providing at least one look-up table containing a 
plurality of coefficients and corresponding values, 
wherein at least some of the look-up table coefficients 
match coefficients in said component images; and 

(C) embedding said watermark in said image by perform- 
ing the following substeps for at least some of said 
component images: (1) selecting a coefficient into 
which a marking value, representative of a correspond- 
ing portion of the watermark, is to be embedded; (2) 
using the value of the selected coefficient to identify a 
corresponding value in said look-up table(s); (3) leav- 
ing the selected coefficient unchanged if the corre- 
sponding value is the same as the marking value; and 
(4) changing the selected coefficient if the correspond- 
ing value is different from the marking value; 

further comprising the step of identifying selected coef- 
ficients as being unembeddable, wherein said unem- 
beddable coefficients are not employed in step (C) to 
embed marking values; and 

further comprising selecting a plurality of threshold 
values, wherein coefficients having values below said 
threshold values are considered small valued and thus 
unembeddable. 

3. A method for applying a digital watermark to an image, 
comprising the steps of: 

(A) deriving from the image a plurality of component 
images, wherein each component image contains coef- 
ficients; 

(B) providing at least one look-up table containing a 
plurality of coefficients and corresponding values, 
wherein at least some of the look-up table coefficients 
match coefficients in said component images; 

(Q shuffling the coefficients prior to said embedding step; 
and 
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(D) embedding said watermark in said image by perform- 
ing the following substeps for at least some of said 
component images: (1) selecting a coefficient into 
which a marking value, representative of a correspond- 
ing portion of the watermark, is to be embedded; (2) 5 
using the value of the selected coefficient to identify a 
corresponding value in said look-up table(s); (3) leav- 
ing the selected coefficient unchanged if the corre- 
sponding value is the same as the marking value; and 
(4) changing the selected coefficient if the correspond- 1Q 
ing value is different from the marking value; 

wherein the image is divided into a plurality of compo- 
nent images of nxm pixels, wherein n and m are 
integers, and wherein each component image is trans- 
formed using the discrete cosine transform (DCT) into 
nxm transform coefficients; wherein the corresponding 15 
values in said look-up table(s) are binary-valued and 
are constrained to have a prescribed maximum run 
length of either binary value; wherein, in the embed- 
ding substep (DX4), the selected coefficient is changed 
minimally to a value having a corresponding value in 20 
the look-up table(s) whose value is the same as the 
marking value, wherein the selected coefficient is 
changed minimally by changing its value to that of the 
nearest coefficient having a corresponding look-up 
table value equal to the marking value. 25 

4. A method as recited in claim 3, wherein, prior to the 
embedding step (C), the image is compressed using JPEG 
compression. 

5. A method as recited in claim 3, further comprising the 
step of shuffling the coefficients prior to said embedding 3Q 
step, wherein said shuffling step comprises concatenating 
the coefficients of a plurality of component images into a 
string and shuffling the order of the coefficients in said 
string; and wherein, after embedding, the string is inverse 
shuffled. 35 

6. A method as recited in claim 3, wherein the marking 
value is embedded multiple times in each component image, 
whereby a majority voting process may be employed to 
decode the respective component images. 

7. A method as recited in claim 3, wherein the marking 4Q 
value is embedded using error correction encoding, and 
wherein error correction decoding is employed to extract the 
marking values. 

8. A method as recited in claim 7, wherein the error 
correction encoding comprises embedding of the marking 45 
value multiple times and the decoding comprises the use of 
majority voting. 

9. A method as recited in claim 3, wherein the marked 
image is decodable to determine whether it has been altered 
and, if so, where in the image such alterations were made. 5Q 

10. A method as recited in claim 3, further comprising the 
step of storing the marked image in a lossy-compression 
form. 

11. A method as recited in claim 3, wherein the method is 
used in a digital camera or camcorder. 55 

12. A method for applying a digital watermark to an 
image, comprising the steps of: 

(A) deriving from the image a plurality of component 
images, wherein each component image contains coef- 
ficients; 60 

(B) providing at least one look-up table containing a 
plurality of coefficients and corresponding values, 
wherein at least some of the look-up table coefficients 
match coefficients in said component images; 

(C) shuffling the coefficients; and 65 

(D) embedding said watermark in said image by perform- 
ing the following substeps for at least some of said 
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component images: (1) selecting a coefficient into 
which a marking value, representative of a correspond- 
ing portion of the watermark, is to be embedded; (2) 
using the value of the selected coefficient to identify a 
corresponding value in said look-up table(s); (3) leav- 
ing the selected coefficient unchanged if the corre- 
sponding value is the same as the marking value; and 
(4) changing the selected coefficient if the correspond- 
ing value is different from the marking value; 

wherein the image is divided into a plurality of compo- 
nent images of nxm pixels, wherein n and m are 
integers, and wherein each component image is trans- 
formed using the discrete cosine transform (DCT) into 
nxm transform coefficients; wherein the corresponding 
values in said look-up table(s) are binary-valued and 
are constrained to have a prescribed maximum run 
length of either binary value; wherein, in the embed- 
ding substep (C)(4), the selected coefficient is changed 
minimally to a value having a corresponding value in 
the look-up table(s) whose value is the same as the 
marking value, wherein the selected coefficient is 
changed minimally by changing its value to that of the 
nearest coefficient having a corresponding look-up 
table value equal to the marking value; and 

further comprising the step of identifying selected coef- 
ficients as being unembeddable, wherein said unem- 
beddable coefficients are not employed in step (C) to 
embed marking values; wherein some of said coeffi- 
cients are identified as DC coefficients that are consid- 
ered unembeddable; and wherein coefficients having 
values below threshold values are considered unem- 
beddable. 

13. A method for applying a digital watermark to an 
image, comprising the steps of: 

(A) deriving from the image a plurality of component 
images, wherein each component image contains coef- 
ficients; 

(B) providing at least one look-up table containing a 
plurality of coefficients and corresponding values, 
wherein at least some of the look-up table coefficients 
match coefficients in said component images; 

(C) shuffling the coefficients prior to said embedding step; 
and 

(D) embedding said watermark in said image by perform- 
ing the following substeps for at least some of said 
component images: (1) selecting a coefficient into 
which a marking value, representative of a correspond- 
ing portion of the watermark, is to be embedded; (2) 
using the value of the selected coefficient to identify a 
corresponding value in said look-up table(s); (3) leav- 
ing the selected coefficient unchanged if the corre- 
sponding value is the same as the marking value; and 
(4) changing the selected coefficient if the correspond- 
ing value is different from the marking value. 

wherein the image is divided into a plurality of compo- 
nent images of size nxm pixels, wherein n and m are 
integers, and wherein each component image is trans- 
formed into coefficients; 

further comprising the step of shuffling the coefficients 
prior to said embedding step; 

wherein said shuffling step comprises concatenating the 
coefficients of a plurality of component images into a 
string and shuffling the order of the coefficients in said 
string; and 

wherein the shuffled string of coefficients are put into a 
plurality of sub-strings and each sub-string is used to 
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embed one watermark or is to be used for error detec- 
tion or for error correction. 
14. A method for applying a digital watermark to an 
image, comprising the steps of: 

(A) deriving from the image a plurality of component 5 
images, wherein each component image contains coef- 
ficients; 

(B) providing at least one look-up table containing a 
plurality of coefficients and corresponding values, 
wherein at least some of the look-up table coefficients 10 
match coefficients in said component images; 

(C) shuffling the coefficients prior to said embedding step; 
and 

(D) embedding said watermark in said image by perform- ^ 
ing the following substeps for at least some of said 
component images: (1) selecting a coefficient into 
which a marking value, representative of a correspond- 
ing portion of the watermark, is to be embedded; (2) 
using the value of the selected coefficient to identify a 
corresponding value in said look-up table(s); (3) leav- 



,775 Bl 

18 

ing the selected coefficient unchanged if the corre- 
sponding value is the same as the marking value; and 
(4) changing the selected coefficient if the correspond- 
ing value is different from the marking value. 

wherein the image is divided into a plurality of compo- 
nent images of size nxm pixels, wherein n and m are 
integers, and wherein each component image is trans- 
formed into coefficients; 

further comprising the step of shuffling the coefficients 
prior to said embedding step; 

wherein said shuffling step comprises concatenating the 
coefficients of a plurality of component images into a 
string and shuffling the order of the coefficients in said 
string; and 

wherein the shuffled string of coefficients are put into a 
plurality of sub-strings and each sub-string is used to 
embed one watermark or is to be used for error detec- 
tion or for error correction. 

***** 
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frequency domain and for determining whether an image so 
watermarked has been tampered with. Watermark insertion 
is accomplished by embedding the bits of a digital signature 
of a hash function of the image in the frequency coefficients 
of the image. Tamper detection is accomplished generally as 
follows: the fragile watermark which was embedded during 
the watermark insertion process is extracted from the image; 
the hash function of the image is computed as in the 
insertion process; it is verified using a public key whether 
the extracted watermark is a valid signature of the hash 
value. If so, then there is assurance that the image has not 
been tampered with. Otherwise, there is reason to conclude 
that the image has been tampered with. 
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FRAGILE WATERMARKS FOR DETECTING 
TAMPERING IN IMAGES 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates generally to a fragile watermarking 
scheme, and more particularly to techniques for inserting 
and extracting fragile watermarks directly in the frequency 
domain of a compressed image and for determining whether 
an image so watermarked has been tampered with. 

2. Description of the Related Art 

A watermark is a digital pattern inserted into a digital 
creation, such as a digital image. The process of inserting a 
watermark into a digital image can be done directly in the 
frequency domain representation typically used in com- 
pressed images. The watermark can be inserted by altering 
certain frequency coefficients while minimizing the result- 
ing distortion. In case of a block-based frequency domain 
representation, such as that used in the JPEG image com- 
pression standard, frequencies as well as blocks can be 
chosen to be altered so that the distortion is minimized. In 
either case, after the watermark has been inserted into the 
image, a procedure, which may be implemented using 
certain software, is typically required to validate the water- 
mark. 

There are different types of watermarks which serve 
different purposes. Tamper-resistant watermarks, for 
example, are designed to identify ownership or the intended 
recipient of a digital image. To function effectively as an 
identifier (i.e., to trace unauthorized distribution of an 
image), a tamper-resistant watermark must be embedded in 
the image so that it is impossible, or at least difficult, to 
remove the watermark without visibly damaging the image. 
Such a watermark must also be resistant to image processing 
techniques, such as cropping, scaling, image enhancement, 
compression/decompression, etc. In addition, a tamper- 
resistant watermark should be readily detectable and recov- 
erable by the proper authorities to permit the tracing and 
identification of the image, even if someone has tampered 
with the image. 

Another type of watermarks, sometimes referred to as 
fragile watermarks, are designed to detect tampering of an 
image. A fragile watermark is embedded in a digital image 
so that, if someone tampers with the image, that tampering 
will modify (or destroy) the watermark. Fragile watermarks 
may be used, for example, in connection with images 
generated by digital cameras to provide a basis for deter- 
mining whether or not an image has been tampered with 
after its creation. 

Various watermarking methods have been proposed. For 
example, U.S. Pat. No. 5,530,759 proposes a digital water- 
mark applied to an original image as a multiplicative cor- 
rection to pixel sample values of the original image in a 
linear color space such that the chromaticities of the pixels 
are not changed. This procedure results in a visible water- 
mark which is simply added to the original image. 

U.S. Pat. No. 5,606,609 sets forth an electronic document 
verification system and method. This refers to a scheme for 
electronically signing documents, but the signature is not 
embedded in the document data itself (i.e., the document 
data is not modified). The signature is just added to the 
document as another field. 

U.S. Pat. Nos. 5,613,004 and 5,687,236 combine stega- 
nography (hiding information that is otherwise in plain 
view) and cryptography (scrambling information that may 
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be sent by unsecured means). Data is watermarked so that if 
it is copied, it is possible to determine who owns the 
original. Thus, the watermarking schemes of these patents 
are of the tamper-resistant type. 

5 U.S. Pat. No. 5,664,018 proposes a watermarking proce- 
dure wherein each of a set of copies of a digitized work has 
a slightly-modified form of a "baseline" watermark that is 
placed within a critical region of the data. This is a tamper- 
resistant scheme that makes it difficult to remove the water- 

10 mark without damaging the content, even if several parties 
with differently watermarked copies of a single image 
collude together in an attempt to remove the watermark. 

U.S. Pat. No. 5,689,587 sets forth a method and apparatus 
for hiding data in images. This is a way to hide information 

15 in images, again to ensure tamper resistance (copyright type 
protection). 

However, none of these patents provide a fragile water- 
marking scheme for detecting tampering. Moreover, none of 

2Q these patents provide a watermarking technique which may 
be implemented directly in the frequency domain represen- 
tation of an image such that the distortion resulting from the 
embedded watermark is minimized. Yet another shortcom- 
ing of these patents is that they do not provide insertion and 

25 verification procedures that work without having to com- 
pletely decompress a compressed image. 

OBJECTS OF THE INVENTION 

Therefore, it is an object of the present invention to 
30 overcome the aforementioned problems. 

It is another object of the present invention to provide a 
scheme for inserting and extracting fragile watermarks in 
frequency domain and for verifying whether an image so 
watermarked has been tampered with. 
35 It is still another object of the invention to provide a 
verification procedure in connection with a fragile water- 
marking scheme for images, whereby, if tampering has 
occurred, the verification procedure will reveal a damaged 
4Q watermark and hence provide a basis for declaring that the 
image has been tampered with. 

It is yet another object of the invention to provide a 
frequency domain watermark insertion and verification pro- 
cess that uses a secret key only in the insertion step while the 
45 verification is carried out using a publicly available key. 
It is yet another object of the invention to provide a 
watermark insertion and verification process that does not 
require full decompression of a compressed image. 

50 SUMMARY OF THE INVENTION 

According to one aspect of the invention, a technique for 
embedding a fragile watermark in a digital image and a 
technique for detecting tampering of a digital image so 
watermarked are provided. 

Another aspect of the invention involves watermarking a 
compressed digital image and detecting tampering of a 
compressed image so watermarked, without having to com- 
pletely decompress the image. 
60 Yet another aspect of the invention involves an insertion 
mechanism that uses a secret key while the corresponding 
verification mechanism uses a publicly available key. 

Watermarking the compressed image initially involves 
computing a hash value for the image which is accomplished 
65 by: partially decoding the compressed digital image to 
generate a plurality of blocks, each block having a plurality 
of transform coefficients; obtaining a quantizer of the high- 
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est frequency coefficient in each block and multiplying that 
coefficient by its quantizer; determining whether to embed a 
watermark bit in each block based on the highest frequency 
transform coefficient in that block and the number of water- 
mark bits remaining to be embedded in the digital image; 
setting to zero the least significant bit (LSB) of the highest 
frequency transform coefficient in each block for which it 
was determined to embed a watermark bit; updating a hash 
value at each block by applying a hash function at each 
block, wherein the hash value computed at the last block is 
a multiple -bit value representative of the entire digital 
image. Once the hash value for the entire image is computed, 
a watermark is computed from the computed hash value 
using a secret key and a digital signature algorithm. Then, 
each watermark bit is embedded in one of the blocks for 
which it was previously determined to do so by setting the 
LSB of the highest frequency transform coefficient in that 
block to match the corresponding watermark bit. 

To determine whether a compressed image so water- 
marked has been tampered with, the following steps are 
performed: partially decoding the compressed watermarked 
image to generate a plurality of blocks, each block having a 
plurality of transform coefficients; determining each block in 
which a watermark bit is embedded; extracting, from each 
block which was previously determined to be an embedder 
of a watermark bit, the LSB of the highest frequency 
transform coefficient in that block to generate an extracted 
watermark; computing a hash value of the digital image by 
applying a hash function at each block based on the zeroed 
value of the LSB of the highest frequency transform coef- 
ficient in that block, wherein the hash value computed at the 
last block is a multiple-bit value representative of the entire 
digital image; applying a digital signature algorithm to the 
extracted watermark and the computed hash value and a 
public key to determine whether the compressed water- 
marked image has been tampered with. 

The watermark insertion procedures can be done directly 
in an image capturing device, such as a digital camera, or 
can be performed by an appropriately configured computer. 
Such a computer can also be used to check a watermarked 
image to determine if tampering has occurred and, if tam- 
pering has occurred, to determine where it occurred. 

Other objects and attainments together with a fuller 
understanding of the invention will become apparent and 
appreciated by referring to the following description and 
claims taken in conjunction with the accompanying draw- 
ings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

In the drawings wherein like reference symbols refer to 
like parts: 

FIG. 1 is a schematic diagram generally showing the 
insertion of a watermark into an image in frequency domain 
in accordance with the invention. 

FIG. 2 is a flow diagram depicting an initial scan proce- 
dure used in connection with a method of inserting a 
watermark into an image in frequency domain in accordance 
with the invention. 

FIG. 3 is a flow diagram depicting a procedure to embed 
bits into an image in connection with a method of inserting 
a watermark into an image in frequency domain in accor- 
dance with the invention. 

FIG. 4 is a schematic diagram showing the verification of 
the existence of the fragile watermark in frequency domain 
from a watermarked image in accordance with the invention. 

FIG. 5 is a flow diagram depicting a method of extracting 
the watermark from an image and verifying its validity to 
determine if the image has been tampered with. 
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FIG. 6 is a block diagram of a digital camera adapted for 
use in connection with the invention. 

FIG. 7 is a block diagram that illustrates the interrela- 
tionship between various components that may be used in 
5 capturing and viewing digital images, as well as processing 
such images in accordance with the invention. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

10 

To watermark an image in the frequency domain, the 
image is scanned once to compute H, which represents a 
k-bit hash value of the image. Using a digital signature 
algorithm, S, and a secret key, K,, the m-bit watermark 

35 W=S(H,KJ is computed. A second pass of the image is then 
made to embed the watermark W into the image. The 
process is illustrated in FIGS. 1, 2 and 3. 

Referring primarily to FIG. 2 and secondarily to FIGS. 1 
and 3, in the initial scan procedure, the hash value H is 

20 initialized to a fixed value in step 101. The compressed 
image data, which may be in the form of a JPEG image I, is 
supplied to a decoder in step 102. The decoder parses the 
headers from the JPEG data, noting the value of q, the 
quantization table entry for the highest frequency 

25 coefficient, which is the 63 rd coefficient for the 8x8 discrete 
cosine transform (DCT) used in JPEG. If there is another 
block of coefficients yet to be decoded and processed (step 
103), the next such block, B,-, is partially decoded in step 
104. Only the entropy coding of the compressed data is 

30 undone, avoiding the de-zig-zagging, dequantization, and 
inverse discrete cosine transform (IDCT) steps needed for 
full decompression. This results in a representation of B t 
made up of only the non-zero quantized coefficients along 
with their locations in the zig-zag order. Since we use the 

35 63 r< * coefficient in special ways, the representation of Bi is 
always made to have the 63^ coefficient even when its value 
is zero. Note that this can be easily done without de-zig- 
zagging the other non-zero quantized coefficients because 
the 63"* coefficient is the last coefficient in the zig-zag order. 

40 The decoded representation of the block B ( - is then passed to 
step 105, where the 63 rd coefficient is multiplied by its 
quantizer, q (obtained in step 102). The reason for this step 
is as follows. Small variations in the higher frequencies are 
invisible to the human eye. The watermark W is embedded 

45 bit-by-bit in the image by modifying only the highest 
frequency coefficient (the 63 rd coefficient in this case) so as 
to minimize the distortion. A watermark bit is embedded 
(later, in step 207, FIG. 3) in a coefficient value by changing 
the LSB of the value to be equal to the watermark bit. If a 

50 quantized 63 rd coefficient has the value v, then changing its 
LSB changes the dequantized coefficient by plus or minus q, 
where q is the quantizer for the 63 rd coefficient. To minimize 
this change, we set the quantizer for the 63 rd coefficient to 
be 1 and replace the 63 coefficient in every block (not just 

55 those embedding watermark bits) directly by their dequan- 
tized values (which arc obtained by multiplying by q). On 
decompression, the distortion can only be plus or minus 1, 
as a result of this step. Since the 63 rd coefficient is typically 
zero in most of the blocks, the increase in compressed size 

60 resulting from the multiplication in step 105 is minimal, as 
only the non-zero coefficients account for most of the 
compressed size. 

A decision is made as to whether a watermark bit is to be 
embedded in block B, in step 106. The decision procedure 

65 for step 106 is designed so as to minimize the resulting 
distortion in the image as well as to minimize the resulting 
size-increase of the compressed image. We denote this 
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decision procedure by the name EMBEDDER-TEST, to 
simplify the subsequent presentation as this decision proce- 
dure is used again in two places. EMBEDDER-TEST is 
fully described as follows. 

For color images, watermark bits are embedded only in 
the luminance plane of the image. This is done so that during 
decompression, when the luminance-chrominances color 
representation is converted back to red, green, and blue pixel 
values (RGB), the resulting distortion is minimized. 
Moreover, the chrominance planes are typically subsampled, 
so any distortion in a single chrominance block results in 
distortions in several RGB blocks. Thus, in grayscale images 
as well as in color images, watermark bits are embedded 
only in the color component numbered zero (which is the 
luminance plane for color images). To minimize the 
distortion, watermark bits are embedded only in the 63 rd 
DCT coefficient, as mentioned previously, To minimize the 
compressed size, only those blocks are chosen to embed a 
watermark bit where the 63rd coefficient is already non-zero. 
This follows from the observation that changing a zero value 
to a non-zero value results in a far greater increase in 
compressed size, compared to changing a non-zero value to 
another nonzero value. However, since EMBEDDER-TEST 
will also be performed by the watermark verification 
procedure, we do not choose as embedders even the blocks 
where the 63 rd coefficient (dequantized) is plus or minus 1, 
as it might potentially be turned to zero on embedding the 
watermark bit, and then the verifier will not be able to decide 
the block to be an embedder. If, at some point, the number 
of watermark bits remaining to be embedded becomes equal 
to the number of blocks remaining in component zero, every 
subsequent block in component zero is decided upon as an 
embedder of a watermark bit. 

If the decision to embed a bit in block B 4 - is yes, then the 
LSB of the 63 rd DCT coefficient is set to zero in step 107 and 
the procedure proceeds to step 108. If the decision is no, then 
the procedure directly proceeds to step 108. The hash value 
H is updated in step 108 using its previous value, and the 
values of all the non-zero quantized coefficients in B ( . along 
with their locations in the zig-zag order, using a one-way 
hash function. The 63 rd coefficient is always included in this 
computation even if it is zero. 

When all the blocks have been processed, the procedure 
proceeds to step 109, where the digital signature algorithm 
S is applied to the computed hash value, H, and the secret 
key, K„ to compute the m-bit watermark W«S(H, K s ). 

Referring primarily to FIG. 3 and secondarily to FIGS. 1 
and 2, in the second pass of the watermark insertion 
procedure, the input JPEG image I is once again fed to a 
decoder which parses its headers noting the value of q, the 
quantizer for the 63"* coefficient, in step 201. The headers 
for the watermarked output JPEG image, 1^,, are produced in 
step 202 by copying them directly from the input image, 
except that the quantizer of the 63"* coefficient in the 
quantizer table is changed to be 1 instead of its previous 
value, q. If there is another block of coefficients yet to be 
decoded and processed (step 203), the next such block, B„ 
is partially decoded in step 204. Again, only the entropy 
coding of the compressed data is undone, avoiding the 
de-zig-zagging, dequantization, and IDCT steps needed for 
full decompression. This results in a representation of B, 
made up of only the non-zero quantized coefficients (except 
for the 63 rd coefficient which is always included in the 
representation) along with their locations in the zig-zag 
order. The 63 rd coefficient of each block is multiplied by the 
q, in step 205. EMBEDDER-TEST is performed in step 206 
to determine whether block B,- is supposed to embed the next 
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bit of W. This determination may be made again on a 
block-by -block basis or may be made using the results of the 
previous decision-making process (step 105), if those results 
are stored in memory. In any case, if block B ( is supposed to 

5 embed the next bit of W, then the LSB of the 63 rd DCT 
coefficient of B, is set to match the next bit of W in step 207 
and the procedure proceeds to step 208. If the decision in 
step 206 is "no", then the procedure directly proceeds to step 
208. In step 208, the coefficients in B, are encoded and 

10 produced as output into the compressed data stream for the 
watermarked image, 1^. Note that the representation of the 
quantized coefficients of B, that we use enables efficient 
encoding as the quantized coefficients are already in the 
zig-zag order, thus avoiding the DCT, quantization, and 

3 5 zig-zagging steps generally required for compression. The 
process repeats until all of the blocks have been processed. 

The verification procedure for the frequency domain 
watermarking method is illustrated in FIGS. 4 and 5 and is 
used to determine if the image has been tampered with. 

20 Referring to FIG. 5, after initializing the hash value H in step 
301, the watermark verification procedure begins to decode 
the input image, 1^, parsing its headers, in step 302. In step 
303, it is determined whether another block remains to be 
decoded. If so, the next block, B„ is partially decoded in step 

25 304. Again, only the entropy coding of the compressed data 
is undone, avoiding the de-zig-zagging, dequantization, and 
IDCT steps needed for full decompression. This results in a 
representation of B, made up of only the non-zero quantized 
coefficients (except for the 63™* coefficient which is always 

30 included in the representation) along with their locations in 
the zig-zag order. EMBEDDER-TEST is performed in step 
305 to determine whether block B,. is supposed to embed the 
next bit of W. If it is, then the LSB of the 63^ coefficient of 
B, is extracted as the next bit of the watermark W in step 

35 306, and then that LSB is set to zero in step 307. The 
procedure moves to step 308 from step 307 as well as from 
step 305 when the block B ( - is not a watermark bit embedder. 
The hash value H is updated in step 308 using its previous 
value, and the values of all the non-zero quantized coeffi- 

40 cients in B,- along with their locations in the zig-zag order, 
using the one-way hash function. The 63 "* coefficient is 
always included in this computation even if it is zero. The 
process continues through all the blocks and in the end, the 
extracted watermark W and the hash value H have been fully 

45 computed. At this point the digital signature verification 
algorithm V (corresponding to the signing algorithm S) is 
applied using the public key (corresponding to the secret 
key KJ to verify whether or not W is the same as S(H, 
in step 309. Note that step 309 does not require the use of 

so the secret key K x . Step 310 examines the output of the digital 
signature verification algorithm V(W,H,Kp) applied in step 
309. If the verification algorithm V(W,H,Kp) is successful, 
then the image has the fragile watermark intact, and thus has 
not been tampered with (step 312). If V(W,H^C p ) outputs 

55 failure, then the fragile watermark (if it ever existed) has 
been destroyed, and it may be concluded that the image has 
been tampered with or never had the fragile watermark 
corresponding to the key pair (K,jy (step 311). 

The effectiveness of the image tampering detection pro- 

60 cess depends on the strength of the hash function and the 
signing and verification algorithms S and V. A variety of 
one-way hash functions may be used, such as a hash 
function called MD5 developed by R. Rivest, or the SHA, or 
RIPEMD hash functions. Similarly, there is considerable 

65 choice available for the signature and verification algorithm 
pair, such as the El Gamal scheme, the DSA algorithm, or 
the RSA algorithm. 
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This fragile watermarking procedure can be modified to storage 26 where decompressed image data may be stored, 

detect roughly the area of the image in which some tarn- The storage block 26 may comprise compact magnetic or 

pering has been done. This is accomplished by dividing the solid-state storage media, either removable or fixed within 

image into some number of regions and applying the whole the camera 20, and may include removable, large-capacity 

insertion procedure separately on each region. Only the 5 PCMCIA-format hard disk cards or flash memory cards, 

regionsthat contain tampering will show a damaged water- ^ emm 2Q mdudes aQal and ^ 2? and 

mark. This modification will not, however, detect a restricted 2$ tivel thro h which im data be transmit . 

form of tampering which is done by formmg an image tha ^ the camera or to external devices. Uncompressed 

is a collage of several regions extracted from different . . , , % . , . 4 . , * « n . 

& * * ■ c • i • u image data may be transmitted, via the analog outputs 27, to 

images or reordered regions from a single image, each in f _ J - n . ' - n & / ' . 

r .r, t i o l * • 10 an LCD screen 29 withm the camera 20, or to external 

region carrying a valid watermark. Such tampering, , . , , , . . . . T , t 

, & . ,., 7 « . ■ „ . • t lL ■ devices such as a VCR or television receiver. Image data, 

however, is likely to be visually obvious, if the regions are , . , , . , ° t 

i h whether compressed or uncompressed, may also be trans- 

, . , , . mit ted through the digital outputs 29 to a digital device such 

Similar fragile watermarking techniques > can be applied in as a ^ ffl wheK . , he ^ cqM ^ dis ]a d or 

the spatial domain as well Instead of the highest-frequency 15 wher(j watermarked ^ could be V6rified 

coefficients, all or some of the pixels can be directly used as , A . . 

watermark bit embedders by setting their LSB to zero prior . FI £- ' » » block diagram that iDusttrtcs the mterrela- 

to the hash computation and then setting that LSB to the ,10n H sm .P be,ween vanou * components that may be used in 

watermark bit in the second pass. Note that this invention ca P tunD 8' P^ssingand viewing digital ullages. One of the 

does not suggest intermingling of the spatial and frequency , 0 more important components 1S a computer system, identified 

domain watermarking processes. It does not suggest that generally by reference numeral 30. The computer system 30 

watermark insertion can be done in the frequency domain as mav be of an y smtable l H* such 35 a mam frame or P e(Sonal 

revealed in this invention while the corresponding water- computer. 

mark verification can be done in the spatial-domain as Computer system 30 comprises a central processing unit 

revealed in this invention. ( cpu ) 31 which ma y be a conventional microprocessor, a 

In addition, the watermark can be chosen to be a visible random access memory (RAM) 32 for temporary storage of 

watermark. A visual watermark can be embedded in the information, and a read only memory (ROM) 33 for per- 

frequency domain by computing the transform frequency manent storage of information. Each of these components is 

coefficients of a distinctive watermarking signal, and simply C0U P^ to a bus 34 ; Operation of the computer system 30 is 

adding the coefficients to those of the image. This process 30 ^P 1 ^ controlled and coordinated by operating system 

works as a result of linearity of the transforms commonly soflware * ™ e optting system, which is embodied in the 

used in image coding (such as DCI), which ensures that s y stem memor y and ™» °" P U 31, coordinates the 

addition in the pixel domain corresponds to addition in the operation of computer system 30 by controlhng allocation of 

frequency domain. A similar visible watermarking process s y stem resources and Performing a variety of tasks, such as 

can be performed in spatial-domain as well. 35 P rocessm g» memor y management, networking and I/O 

It should be noted that the block and flow diagrams used rations, among others, 

to illustrate the watermark insertion, extraction and verifi- Also coupled to bus 34 by a controller 35 is a diskette 

cation procedures of the present invention, illustrate the drive 36 int0 wtuch a non-volatile mass storage device such 

performance of certain specified functions and relationships as a diskette 37 ma y be inserted. Similarly, a controller 38 

thereof. The boundaries of these functional blocks have been 40 interfaces between bus 34 and a compact disc (CD) ROM 

arbitrarily defined herein for the convenience of description. drive 39 which » ada P ted to receive a CD ROM 40. A hard 

Alternate boundaries may be defined so long as the specified disk 41 15 Provided as part of a fixed disk drive 42 which is 

functions and relationships thereof are appropriately formed. C0U P led to bus 34 b y a dlsk controller 43. 

Moreover, the flow diagrams do not depict syntax or any Software for the watermarking techniques may be stored 

particular programming language. Rather, they illustrate the 45 on slora S e devices 207 and 210 and transferred to CPU 31 

functional information one skilled in the art would require to for execution. Alternatively, the software may be stored in 

fabricate circuits or to generate software to perform the RAM 32 or ROM 33. Similarly, image data be loaded into 

processing required. Each of the functions depicted in the and extracted from computer system 30 using removable 

block and flow diagrams may be implemented, for example, storage media devices such as the diskette 37 and CD ROM 

by software instructions, a functionally equivalent circuit 50 4 ^- 

such as a digital signal processor circuit, an application Image data may be input into computer system 30 in other 

specific integrated circuit (ASIC) or combination thereof. ways as well. Film-based images 44 generated by a film 

The watermarking techniques of the present invention camera 45 can be digitized by a scanner 46 for storage and 

may be employed in connection with various devices includ- processing by the computer 30. The digital camera 20 can 

ing a digital camera, a block diagram of which is illustrated 55 directly digitize images and transmit them to the computer 

in FIG. 6. Operating under microprocessor control, the 30, as explained above. A keyboard 51 and mouse 52, which 

digital camera 20 has a charge-coupled device (CCD) image are coupled to bus 34 via a controller 53, facilitate the input 

sensor that captures an image and converts it to an analog of suc h data and otherwise provide a means for entering 

electrical signal in block 21. The analog signal is then information into computer system 30. 

processed and digitized in block 22, after which the digital 60 Image data may also be transferred to and from computer 

image is temporarily stored in a frame buffer 23 while it 30 for remote locations. To this end, computer 30 may also 

undergoes digital processing in block 24. The digital image include a communications adapter 54 which enables the 

processing block 24 performs several functions including computer 30 to communicate with networks 55, which may 

compression and decompression. Processing block 24 may include local area networks (LANs), the internet or online 

also perform the watermarking techniques of the present 65 services, via direct connections or via modem, 

invention using hardware or software. Under user control In accordance with the invention, images that have been 

25, the processing block 24 interfaces with in-camera image previously watermarked, say, in the digital camera 20 may 
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be transmitted to computer 30 for verification. Alternatively, 
unmarked images may be watermarked and later verified in 
computer 30 using appropriate hardware or software that is 
executed by the CPU 31. 

Digital images transmitted or stored in computer 30 may 5 
be viewed in a number of different ways. A printer 56 
attached to computer 30 can produce color prints that vary 
in quality depending on the printer 56. Another option is to 
view the images on a display 57 associated with the com- 
puter 30. Yet another choice is to display the images on a 30 
television monitor using a VCR. 

While the invention has been described in conjunction 
with specific embodiments, it will be evident to those skilled 
in the art in light of the foregoing description that many 
further alternatives, modifications and variations are pos- 15 
sible. For example, visible watermarking process and the 
invisible watermarking process described in this invention 
can be combined depending upon the application. 
Furthermore, the choice of blocks to be used in the water- 
marking process can be made application dependent. Thus, 20 
the invention described herein is intended to embrace all 
such alternatives, modifications, applications and variations 
as may fall within the spirit and scope of the appended 
claims. 

25 

What is claimed is: 

1. A method of watermarking a compressed digital image, 
comprising the steps of: 

partially decoding the compressed digital image to gen- 
erate a plurality of data blocks, each block having a 3Q 
plurality of transform coefficients; 

determining whether to embed a watermark bit in each 
block based on one of the plurality of transform coef- 
ficients in that block and a number of watermark bits 
remaining to be embedded in the digital image; 35 

computing a watermark, having a plurality of bits, for the 
entire digital image by applying a digital signature 
algorithm and a secret key; and 

embedding a watermark bit in each block for which it was 
determined to do so in said determining step by setting 40 
the one of the plurality of transform coefficients in that 
block to match a corresponding bit of the computed 
watermark. 

2. The method of claim 1, wherein the one of the plurality 
of transform coefficients in each block is the transform 45 
coefficient representative of the highest vertical frequency 
and highest horizontal frequency in that block. 

3. The method of claim 1, wherein said step of computing 
the watermark further comprises the step of: 

setting to zero at least one bit of the one of the plurality 50 
of transform coefficients in each block for which it is 
determined in said determining step to embed a water- 
mark bit. 

4. The method of claim 3, further comprising the step of: 
encoding the plurality of blocks after said embedding step 

is completed to re-compress the watermarked image. 

5. The method of claim 3, wherein said of computing the 
watermark further comprises the step of: 

updating a hash value at each block by applying a hash 60 
function at each block, wherein the hash value com- 
puted at a last block is the multiple-bit hash value 
representative of the entire digital image. 

6. The method of claim 5, wherein said step of computing 
the watermark further comprises the steps of: 65 

obtaining a quantizer of the one of the plurality of 
transform coefficients in each block; and 
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multiplying the one of the plurality of transform coeffi- 
cients in each block by its quantizer. 

7. A method of watermarking a compressed digital image, 
comprising the steps of: 

computing a watermark, having a plurality of bits, for the 

entire digital image, comprising the steps of: 

partially decoding the compressed digital image to 
generate a plurality of data blocks, each block having 
a plurality of transform coefficients; 

obtaining a quantizer of the one of the plurality of 
transform coefficients in each block; 

multiplying the one of the plurality of transform coef- 
ficients in each block by its quantizer; 

determining whether to embed a watermark bit in each 
block based on one of the plurality of transform 
coefficients in that block that is representative of a 
highest vertical frequency and a highest horizontal 
frequency in that block and a number of watermark 
bits remaining to be embedded in the digital image; 

setting to zero at least one bit of the one of the plurality 
of transform coefficients in each block for which it is 
determined in said determining step to embed a 
watermark bit; 

updating a hash value at each block by applying a hash 
function at each block, wherein the hash value com- 
puted at a last block is a multiple-bit hash value 
representative of the entire digital image; and 

applying a secret key and a digital signature algorithm 
to the multiple-bit hash value to compute the water- 
mark; and 

embedding a watermark bit in each block for which it was 
determined to do so in said determining step by setting 
the at least one bit of the one of the plurality of 
transform coefficients in that block to match a corre- 
sponding bit of the computed watermark. 

8. A method of detecting tampering of a compressed 
watermarked image, comprising the steps of: 

partially decoding the compressed watermarked image to 
generate a plurality of data blocks, each block having 
a plurality of transform coefficients; 

determining each block in which a watermark bit is 
embedded; 

extracting, from each block for which it was determined 
in said determining step to be an embedder of a 
watermark bit, at least one bit of one of the plurality of 
transform coefficients in that block to generate an 
extracted watermark; 

computing a hash value of the digital image by applying 
a hash function at each block based on a zeroed value 
of the at least one bit of one of the plurality of transform 
coefficients in that block, wherein the hash value com- 
puted at a last block is a multiple-bit value represen- 
tative of the entire digital image; and 

applying a digital signature verification algorithm to the 
extracted watermark and the multiple-bit computed 
hash value and a public key to determine whether the 
compressed watermarked image has been tampered 
with. 

9. The method of claim 8, wherein the one of the plurality 
of transform coefficients in each block is the transform 
coefficient representative of the highest vertical frequency 
and highest horizontal frequency in that block. 

10. An image capturing device, comprising: 

a sensor for capturing light and converting the light into 

an analog image signal; 
an analog-to-digital converter for converting the analog 

image signal to a digital image; and 
a digital image processor for compressing the digital 

image to generate a plurality of data blocks, each block 
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having a plurality of transform coefficients, determin- 
ing whether to embed a watermark bit in each block 
based on one of the plurality of transform coefficients 
in that block and a number of watermark bits remaining 
to be embedded in the digital image, computing a 5 
watermark, having a plurality of bits, for the entire 
digital image, and embedding a watermark bit in each 
block for which it was determined to do so by setting 
the one of the plurality of transform coefficients in that 
block to match a corresponding bit of the computed 
watermark. 10 

11. The image capturing device of claim 10, wherein said 
digital image processor sets to zero at least one bit of the one 
of the plurality of transform coefficients in each block for 
which it is determined to embed a watermark bit. 

12. The image capturing device of claim 11, wherein said 
digital image processor encodes the plurality of blocks after 
the watermark bit embedding is completed to re-compress 
the watermarked image. 

13. The image capturing device of claim 11, wherein said 
digital image processor updates a hash value at each block 
by applying a hash function at each block, wherein the hash 20 
value computed at a last block is the multiple-bit hash value 
representative of the entire digital image. 

14. The image capturing device of claim 11, wherein said 
digital image processor obtains a quantizer of the one of the 
plurality of transform coefficients in each block and mul- 25 
tiples the one of the plurality of transform coefficients in 
each block by its quantizer. 

15. A computer system including a processor and a 
memory having a computer-readable readable program code 
embodied therein for causing the processor to detect tam- 
pering of a compressed watermarked digital image by per- 30 
forming the steps of: 

partially decoding the compressed watermarked image to 
generate a plurality of data blocks, each block having 
a plurality of transform coefficients; 

determining each block in which a watermark bit is 35 
embedded; 

extracting, from each block for which it was determined 
in said determining step to be an embedder of a 
watermark bit, at least one bit of one of the plurality of 
transform coefficients in that block to generate an 40 
extracted watermark; 

computing a hash value of the digital image by applying 
a hash function at each block based on a zeroed value 
of the at least one bit of one of the plurality of transform 
coefficients in that block, wherein the hash value com- 45 
puted at a last block is a multiple -bit value represen- 
tative of the entire digital image; and 

comparing the extracted watermark with the multiple-bit 
value by applying a digital signature verification algo- 
rithm to determine whether the compressed water- 50 
marked image has been tampered with. 

16. The computer system of claim 15, wherein the one of 
the plurality of transform coefficients in each block is the 
transform coefficient representative of the highest vertical 
frequency and highest horizontal frequency in that block. 5S 

17. A method of watermarking a digital image, having a 
plurality of pixels, comprising the steps of: 

computing a watermark, having a plurality of bits, for the 
entire digital image, comprising the steps of: 
determining whether to embed a watermark bit in each 
pixel based on one of a plurality of bits representa- 60 
tive of that pixel and a number of watermark bits 
remaining to be embedded in the digital image; 
setting to zero at least one bit of each pixel for which 
it is determined in said determining step to embed a 
watermark bit; 65 
updating a hash value at each pixel by applying a hash 
function at each pixel, wherein the hash value com- 



puted at a last pixel is a multiple-bit hash value 
representative of the entire digital image; and 
applying a secret key and a digital signature algorithm 
to the multiple-bit hash value to compute the water- 
mark; and 

embedding a watermark bit in each pixel for which it was 
determined to do so in said determining step by setting 
the at least one bit of that pixel to match a correspond- 
ing bit of the computed watermark. 

18. The method of claim 17, wherein the at least one bit 
is the least significant bit. 

19. A method of detecting tampering of a watermarked 
digital image, having a lo plurality of pixels, comprising the 
steps of: 

determining each pixel in which a watermark bit is 
embedded; 

extracting, from each pixel for which it was determined in 
said determining step to be an embedder of a water- 
mark bit, at least one of a plurality of bits representative 
of that pixel to generate an extracted watermark; 

computing a hash value of the digital image by applying 
a hash function at each pixel based on a zeroed value 
of the at least one bit of that pixel, wherein the hash 
value computed at a last pixel is a multiple-bit value 
representative of the entire digital image; and 

applying a digital signature verification algorithm to the 
extracted watermark and the multiple-bit computed 
hash value and a public key to determine whether the 
compressed watermarked image has been tampered 
with. 

20. The method of claim 19, wherein the at least one bit 
is the least significant bit. 

21. A method of watermarking a compressed digital 
image, comprising the steps of: 

adding a visible watermark; 

computing an invisible watermark from the digital image 
and the visible watermark, comprising the steps of: 
partially decoding the compressed digital image to 
generate a plurality of data blocks, each block having 
a plurality of transform coefficients; 
obtaining a quantizer of the one of the plurality of 

transform coefficients in each block; 
multiplying the one of the plurality of transform coef- 
ficients in each block by its quantizer; 
determining whether to embed an invisible watermark 
bit in each block based on one of the plurality of 
transform coefficients in that block that is represen- 
tative of a highest vertical frequency and a highest 
horizontal frequency in that block and a number of 
invisible watermark bits remaining to be embedded 
in the digital image; 
setting to zero at least one bit of the one of the plurality 
of transform coefficients in each block for which it is 
determined in said determining step to embed an 
invisible watermark bit; 
updating a hash value at each block by applying a hash 
function at each block, wherein the hash value com- 
puted at a last block is a multiple-bit hash value 
representative of the entire digital image; and 
applying a secret key and a digital signature algorithm 
to the multiple-bit hash value to compute the invis- 
ible watermark; and 
embedding an invisible watermark bit in each block for 
which it was determined to do so in said determining 
step by setting the at least one bit of the one of the 
plurality of transform coefficients in that block to match 
a corresponding bit of the computed invisible water- 
mark. 
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